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PREFACE 

One  of  the  major  missions  of  the  Rome  Air  Development 
Center  (RADC)  is  the  development  of  procedures  and  techniques  for 
improving  the  readiness  and  supportability  of  weapon  systems.  In 
support  of  this  mission,  RADC  has  sponsored  a  myriad  of  studies, 
analyses,  and  developments  that  have  resulted  in  techniques,  standards, 
and  procedures  aimed  at  reaching  this  goal. 

In  the  1980s  all  the  military  services  have  recognized  the 
importance  of  improving  the  diagnostic  capability  of  weapon  systems  as  a 
means  for  rapid  troubleshooting  and  repair  of  these  systems.  The 
research  and  development  efforts  conducted  by  RADC  are  reflected  in 
this  guide  by  synthesizing  the  results  of  these  many  efforts  and  filling 
gaps  to  provide  both  government  and  industry  with  a  compendium  of 
procedures  and  techniques  which  may  be  used  to  improve  the  fielded 
weapon  systems’  diagnostic  capability. 

Many  other  programs  have  made  the  contributions  that  are 
included  in  these  guides.  Information  has  been  freely  included  from 
various  military  service  and  industry  work.  Among  these  is  the  Air 
Force’s  Generic  Integrated  Maintenance  Diagnostics  Program  (GIMADS). 
The  Navy’s  Integrated  Diagnostic  Support  System  (IDSS)  and  the  Army’s 
Integrated  Diagnostics  In  the  Maintenance  Environment  (AIDME)  have 
also  made  valuable  contributions  to  this  guide,  in  this  manner,  material 
from  all  of  the  other  service  organizations  is  now  available  for  Joint 
Service  use. 

Three  (3)  guides  have  been  written  which  are  aimed  at  the 
following  users: 

o  Government  Program  Manager 

o  Contractor  Program  Manager 

o  System  Designer. 

Thus,  the  guidance  material  required  by  a  specific  user  will  be  included  in 
one  of  these  three  (3)  guides. 

It  is  believed  that  this  guidance  material  represents  a 
comprehensive  look  at  the  problems  in  fielding  a  satisfactory  diagnostic 
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capability  and  a  structured  system  engineering  approach  to  solving  these 
problems.  RADC  solicits  comments  on  this  guidance  material,  as  a 
means  for  improvements  in  the  coming  years. 

These  guides  have  been  prepared  under  contract  by  Giordano 
Associates,  Inc.,  with  subcontractor  assistance  from  Grumman 
Aerospace  Corporation  and  Rockwell  International. 
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INTRODUCTION 


WHY  ALL  THIS  WORRY  ABOUT  DIAGNOSTICS? 

Let’s  put  the  diagnostic  problem  in  its  proper  perspective. 
You’ve  got  a  problem  with  your  automobile  and  you  turn  to  a  mechanic 
for  help.  Historically,  you  realize  that  the  problem  may  be  fixed  or  indeed, 
for  some  reason,  you  may  have  to  go  back  one  or  more  times  before  the 
problem  is  corrected,  or  you  give  up.  We  are  talking  about  automobiles. 
Automobiles,  which  a  manufacturer  has  produced  tens  of  thousands  of 
times,  have  a  historical  record  of  their  reliability  and  maintainability  and 
have  been  redesigned  and  reengineered  many  times. 

When  comparing  your  automobile  to  an  extremely  complex 
weapon  system  that  is  pushing  the  state  of  the  art  and  produced  in 
limited  numbers,  with  questionable  historical  data  on  their  operation,  one 
can  easily  understand  the  magnitude  of  the  problem. 

It  is  not  the  purpose  of  this  guide  to  provide  a  comprehensive 
discussion  of  the  diagnostic  problems,  but  rather  to  furnish  guidance  for 
government  and  industry  people  in  circumventing  known  problem  areas. 
However,  to  understand  the  magnitude  of  the  problem  a  few  examples 
follow. 


In  one  six-month  period,  at  one  F-16  tactical  fighter  wing,  over 
13,600  maintenance  manhours  were  reported  for  the  processing  of 
unnecessary  removals.  This  equals  about  20  people  just  working  on 
troubleshooting  these  "good"  items. 

A  DoD  Task  Force  on  Productivity  in  Support  Operations 
(1986)  found  that  20  to  50  percent  of  avionics  maintenance  actions 
resulted  in  removal  of  items  with  no  evidence  of  failures. 

The  deployment  of  an  avionics  Intermediate  shop  for  fighter 
aircraft  to  a  remote  location  can  require  anywhere  from  three  to  1 1  C- 
141B  equivalent  loads.  In  wartime,  there  just  will  not  be  enough  cargo 
aircraft  to  respond  to  this  need.  In  peacetime,  it's  just  plain  costly. 

The  diagnostic  problem  is  not  unique  to  any  one  service,  nor  to 
any  one  type  of  weapon  system.  It  manifests  itself  throughout  the  military 
services.  The  problem  can  be  an  engineering  or  a  field  problem.  It  can 
be  a  man  or  a  machine  problem.  It  can  be  a  wartime  or  a  peacetime 
problem.  It  can  be  a  prime  system  or  a  supportability  problem.  The 
problem  manifests  itself  in  different  ways  for  different  types  of  weapon 
systems,  but  the  consequences  are  all  the  same— long  times  to 
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troubleshoot,  removal  of  items  which  have  not  failed,  long  logistic  tails, 
and  an  overall  lack  of  confidence  in  the  entire  diagnostic  capability. 
Obviously,  the  result  is  lack  of  readiness  and  a  waste  of  dollars  and 
manpower. 

There  are  a  multitude  of  reports  which  adequately  describe  the 
problem.  Two  of  these  reports  give  a  comprehensive  picture  of  the 
problem  and  possible  solutions.  These  are: 

"Isolation  of  Faults  in  Air  Force  Weapon  and  Support  Systems," 
Committee  on  Isolation  of  Faults  In  Air  Force  Weapon  and  Support 
Systems,  Air  Force  Studies  Board,  Commission  on  Engineering  and 
Technical  Systems,  National  Research  Council. 

"Report  for  the  Department  of  Defense  orrthe  Implementation 
of  Integrated  Diagnostics,"  prepared  by  the  National  Security  Industrial 
Association’s  Integrated  Diagnostics  Working  Group,  September  1 984. 


HISTORICALLY  THE  FIELDED  DIAGNOSTIC  CAPABILITY  HAS  NOT 
UVED  UP  TO  THE  PROMISES 


Government  and  industry  must  share  the  responsibility  for  what 
has  happened  in  the  past.  On  the  government’s  side,  there  tends  to  be  a 
lack  of  knowledge  on  how  to  specify  what  is  needed  and  how  to  make 
sure  the  government  gets  what  it  needs.  On  the  contractor’s  side  is  a 
lack  of  understanding  of  the  importance  of  fielding  a  satisfactory 
diagnostic  capability  and  still  maintain  schedule  and  cost  limitations. 
Hopefully,  the  series  of  guides  produced  under  this  program  will  help  to 
alleviate  this  situation. 

The  military  services,  as  well  as  the  Office  of  the  Secretary  of 
Defense,  understand  the  urgency  of  this  problem  and  have  established 
multimillion  dollar  programs  to  help  alleviate  this  situation,  both  from  a 
technology  and  a  management  perspective.  For  the  most  part,  these 
programs  are  generic-applicable  to  a  variety  of  weapon  systems. 


DIAGNOSTIC  IMPROVEMENT  PROGRAMS  ARE  UNDERWAY 
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HOW  CAN  THIS  GUIDE  HELP  YOU? 

The  outputs  from  many  of  the  government-sponsored 
programs  are  a  variety  of  techniques,  procedures,  standards,  and 
devices  which  can  be  applied  to  the  acquisition  of  a  system’s  diagnostic 
capability.  However,  this  type  of  information  appears  in  a  variety  of 
reports,  military  standards,  specifications,  and  other  documents.  The 
major  focus  of  this  guide  is  to  bring  together  this  knowledge  in  a  usable 
form  and  tie  this  to  the  various  diagnostic  activities  which  occur  during 
the  acquisition  and  deployment  of  a  weapon  system.  In  addition,  where 
holes  exist  in  this  acquisition  process,  the  guide  attempts  to  fill  them. 
Following  this  procedure  will  help  you  in  doing  a  better  job  of  acquiring  a 
weapon  system  diagnostic  capability. 

This  guide  is  for  THE  CONTRACTOR  PROGRAM  MANAGER 
-to  help  him  to  prepare  an  adequate  response  to  a  government  RFP  and 
to  properly  manage  the  development  of  the  diagnostic  capability  for  a 
weapon  system,  once  a  contract  is  underway.  This  is  the  second  in  a 
series  of  three  guides.  The  first  guide  is  a  Government  Program 
Manager’s  Guide  which  helps  him  to  specify  the  required  diagnostic 
capability  and  take  the  steps  to  assure  that  the  requirements  are  met. 
The  third  guide  is  a  comprehensive  Design  Encyclopedia,  which  provides 
detailed  methods,  procedures,  tools,  and  trade-off  information  which  can 
be  applied  to  the  design  and  demonstration  of  a  weapon  system’s 
diagnostic  capability.  This  third  guide  is  aimed  at  the  designer  and 
analyst. 


THREE  GUIDES  -  ONE  FOR  EACH  TYPE  OF  USER.  THIS  ONE  IS 
FOR  THE  CONTRACTOR  PROGRAM  MANAGER 


WHAT  ARE  THE  MAIN  THRUSTS  OP  THIS  GUIDE? 

Historically,  the  development,  test,  and  evaluation  of  the 
diagnostic  capability  for  a  weapon  system  often  have  been  more  or  less 
an  afterthought.  For  many  seemingly  logical  reasons,  satisfying  prime 
system  "performance"  requirements  has  taken  precedence  over 
diagnostic  requirements.  Simply,  the  goal  of  this  guide  is  to  promote  a 
realization  that  diagnostic  requirements  are  indeed  system  performance 
requirements,  and  thus  are  an  integral  part  of  weapon  systems' 
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development,  test,  and  evaluation.  Thus  this  guide  has  the  following 
attributes  in  relation  to  the  development  of  this  diagnostic  capability: 


o  Describes  a  system  engineering  approach 

o  Gives  guidance  for  the  preparation  of  diagnostic  portions  of 
proposals  and  specifications 

o  Delineates  a  usable  diagnostic  allocation  procedure 

o  Describes  a  design  procedure  for  the  integration  of  all 
diagnostic  elements 

o  Suggests  organizational  responsibilities  and  relationships 
be  established  for  development  of  the  entire  diagnostic 
capability 

o  Encourages  concurrent  design,  test,  and  evaluation  of  the 
entire  diagnostic  capability  with  the  prime  hardware  and 
software 

o  Allows  for  the  a  comprehensive  maturation  period  to  meet 
diagnostic  requirements 


A  SYSTEMS  ENGINEERING  APPROACH  IS  KEYI 


DEFINITIONS 


WHAT  DO  WE  MEAN  BY  INTEGRATED  DIAGNOSTICS? 

Before  using  this  guide  it  is  imperative  that  you  understand  the 
definition  of  a  few  words.  The  first  term  is  "testability, n  which  is  defined 
as  "a  design  characteristic  which  allows  the  status  (operable,  inoperable, 
or  degraded)  of  an  item  to  be  confidently  determined  and  the  isolation  of 
faults  within  the  item  to  be  performed  in  a  timely  manner.”  Therefore, 
testability  may  be  regarded  as  inherent  to  the  item’s  design. 

"Diagnostics"  is  defined  as  "the  hardware,  software  and/or 
other  documented  means  used  to  determine  a  malfunction  has  occurred 
and  to  isolate  the  cause  of  the  malfunction.”  It  also  refers  to  ”the  action 
of  detecting  and  isolating  failures.” 

"Integrated  diagnostics"  is  defined  as  a  "structured  design  and 
management  process  to  achieve  the  maximum  effectiveness  of  a 
weapon  system’s  diagnostic  capability  by  considering  and  integrating  all 
related  pertinent  diagnostic  elements."  The  process  includes  interfaces 
between  design,  engineering,  testability,  reliability,  maintainability, 
human  engineering,  and  logistic  support  analysis.  The  goal  is  a  cost- 
effective  capability  to  detect  and  unambiguously  isolate  all  faults  known 
or  expected  to  occur  In  weapon  systems  and  equipment  in  order  to 
satisfy  weapon  system  mission  requirements. 

"Diagnostic  capability"  refers  to  all  the  capabilities  associated  with  the 
detection  and  isolation  of  faults,  including  automatic  and  manual  testing, 
personnel,  training,  maintenance  aiding,  and  technical  information. 

"Diagnostic  element"  is  defined  as  one  part  of  the  diagnostic  capability 
(e.  g.,  ATE). 

"Diagnostic  Subsystem"  is  defined  as  all  the  diagnostic  elements, 
which  constitute  a  weapon  system's  diagnostic  capability. 

"Embedded  diagnostics"  is  defined  as  any  portion  of  the  weapon 
system’s  diagnostic  capability  which  is  an  integral  part  of  the  prime 
system  or  support  system.  "Integral”  implies  that  the  embedded  portion 
is  physically  enclosed  in  the  prime  system  and/or  permanently  attached- 
•physically  or  electrically. 

"External  diagnostics"  is  defined  as  any  portion  of  the  weapon 
system’s  diagnostic  capability  which  is  not  embedded. 
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USE 
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For  a  better  understanding  of  the  various  diagnostic  activities 
that  take  place  during  the  acquisition  and  deployment  of  a  weapon 
system,  a  Roadmap  has  been  prepared  for  your  use.  The  Roadmap 
depicts  all  of  the  diagnostic  activities  that  take  place  during  each  phase 
of  weapon  system  acquisition  and  deployment.  The  Roadmap  is  shown 
in  Figure  1 ,  with  inputs  and  outputs  for  each  activity.  This  Roadmap 
gives  the  reader  the  entire  picture  of  both  government  and  industry 
diagnostic  activities  which  are  aimed  at  developing  an  adequate 
diagnostic  capability.  It  is  recognized  that  there  is  no  single  Roadmap 
that  can  apply  to  all  situations.  Thus  the  Roadmap  is  designed  with 
multiple  entry  points  to  provide  flexibility. 


THE  ROADMAP  GIVES  YOU  THE  BIG  PICTURE 


The  structure  of  the  guide  is  built  around  this  Roadmap.  The 
activities  on  the  Roadmap  relate  to  seven  basic  requirements  listed  in 
Table  1.  This  document  is  a  structured  accordingly  to  these  seven 
requirements. 

Reference  to  a  specific  requirement  is  shown  on  the 
Roadmap,  so  the  reader  can  quickly  relate  a  diagnostic  activity  on  the 
Roadmap  to  specific  guidance  information  contained  in  this  guide. 


TABLE  1.  ESTABLISHED  REQUIREMENTS 


REQUIREMENT# 

REQUIREMENT 

1 

ESTABLISHING  AND  JUSTIFYING  A  PROGRAM 

FOR  ACQUIRING  A  DIAGNOSTIC  CAPABILITY 

2 

ESTABLISHING  AND  ALLOCATING  DIAGNOSTIC 
REQUIREMENTS 

3 

DESIGNING  THE  DIAGNOSTIC  CAPABILITY 

4 

ANALYSIS  AND  ASSESSMENT  OF  THE  PERFORMANCE 

OF  THE  DIAGNOSTIC  CAPABILITY 

5 

CONDUCTING  DESIGN  REVIEWS 

6 

CONDUCTING  TEST  AND  EVALUATION 

7 

MATURATION  OF  THE  DIAGNOSTIC  CAPABILITY 

Each  one  of  these  basic  requirements  is  foffowed  by  detailed 
requirements  (e.  g.,  Requirement  1.2,  Preparing  an  RFP/SOW/ 
Specification).  Each  of  these  detailed  requirements  is  tied  to  a  weapon 
system  activity  and  a  weapon  system  acquisition  phase. 

The  three  guides  are  structured  in  essentially  the  same  way- 
-the  difference  being  the  guidance  material  supplied  is  tailored  for  the 
USER  of  each  specific  guide.  Each  requirement  is  color  coded  or 
highlighted  for  easy  access  to  the  information  the  user  requires. 

Each  of  the  guides  contains  a  Lessons  Learned  Appendix, 
(Appendix  A)  which  will  help  the  user  to  understand  how  this  guidance 
information  applies  to  real-world  acquisitions.  Appendix  B  lists  the 
acronyms  used. 
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I  nos  tic  Activity  Roadmap 


It  is  recognized  that  a  guide  of  this  type  cannot  contain  all  the 
necessary  information  that  the  user  requires.  In  these  cases,  an  attempt 
has  been  made  to  cite  reference  documents,  such  as  military  standards, 
handbooks,  and  reports. 


AN  AUTOMATED  VERSION  OF  THIS  GUIDANCE 
PLUS  SOME  COMPUTERIZED  TOOLS  ARE  AVAILABLE 


To  aid  in  the  use  of  these  guides,  a  computerized,  interactive 
version  of  of  all  these  guides  has  been  developed.  In  addition,  a  number 
of  computer-aided  tools,  suitable  for  inclusion  in  CAD  modules,  have 
been  included  for  use  by  design  engineers  in  the  implementation  of 
techniques,  procedures,  etc.  These  are  contained  In  the  Design 
Encyclopedia.  If  you  are  interested  in  obtaining  these  software  programs, 
you  may  contact  Rome  Air  Development  Center,  RADC/RBET,  Griffiss 
Air  Force  Base,  New  York,  13441-5000,  (telephone  number: 
31 5-330-4726,  AV  587-4723). 


PROGRAMMATIC 


REQUIREMENT  #1 


ESTABLISHING  AND  JUSTIFYING  A  PROGRAM  FOR  ACQUIRING  A  DIAGNOSTIC 
CAPABILITY 

OVERVIEW 

DoD  organizations  are  often  disappointed  at  the 
performance  of  a  weapon  system’s  diagno  c  capability, 
once  it  is  deployed.  This  disappointment  often  results  in 
frustration  by  the  user,  an  adversarial  relationship 
between  the  acquirer  and  the  producer,  and  costly 
engineering  changes.  The  fact  is  that  the  quality  of  the 
acquired  diagnostic  capability  is  a  two-way  street.  In  the 
case  of  the  government,  careful  consideration  must  be 
given  to  what  you  want  the  contractor  to  deliver.  In  the 
case  of  the  contractor,  he  must  be  dedicated  to 
producing  a  quality  product.  Specifying  fault  detection 
and  isolation  requirements  is  a  difficult,  complex  job  for 
the  Government  Program  Manager.  Justifying  his 
program  to  a  higher  authority  in  clear,  concise  terms  is 
essential.  Establishing  realistic  and  feasible  plans  for 
satisfying  these  requirements  is  a  prime  responsibility  of 
the  Contractor  Program  Manager.  Implementing  these 
plans  is  the  responsibility  of  the  weapon  system 
designer.  Without  a  clear  understanding  and  close 
cooperation  among  these  people,  production  of  a  less- 
than-satisfactory  diagnostic  capability  is  inevitable. 


IMPORTANT  CONSIDERATIONS  TO  BE  ADDRESSED 

Reqmt 

1.1  Review  the  Statement  of  Need  (SON)  to  assure  a  clear  understanding  of 
the  basis  for  the  development  program. 

1.2  Diagnostic  considerations  are  a  very  important  part  of  your  proposal. 

1.3  Tie  diagnostic  capability  plans  to  the  system  engineering  plans. 

1.4  Make  sure  that  specific  information  on  diagnostic  and  capability  Issues 
are  available  for  Inclusion  In  SCPs  and  DCPs. 
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REVIEWING  A  STATEMENT  OF  NEED 


REQUIREMENT  #1.1 


WEAPON 
SYSTEM 
ACQ.  PHASE 

OPER. 

REQMTS. 

CONCEPT 

EXPL0R. 

DEM/ 

VAL 

FSD 

PROD/ 

DEPL. 

WEAPON 

SYSTEM 

ACTIVITIES 

a a a 

RFP  RFP  RFP 

PREP  PREP  PREP 

DIAGNOSTIC 

ACTIVITIES 

A  A  A  A 

SON  REVIEWED 

DIAGNOSTIC  ACTIVITY 

The  major  consideration  in  the  initiation  of  a  weapon  system  acquisition  program 
is  the  preparation  of  a  Statement  of  Need.  Each  of  the  services  has  its  own  designation 
for  this  document.  For  a  major  weapon  system,  DoD  Instruction  5000.2  designates  this 
document  as  a  "Mission-Need  Statement  (MNS)."  For  less-than-major  systems,  the 
services  use  other  terms,  such  as  "Operational  Requirements  Document"  and  "Required 
Operational  Capability."  It  is  important  that  these  documents  reflect  these  operational 
requirements  in  terms  which  can  be  properly  interpreted  to  produce  diagnostic 
requirements.  The  contractor  is  not  involved  in  the  generation  of  this  document  but  must 
be  aware  of  it’s  content  since  it  forms  the  baseline  upon  which  the  diagnostic  requirements 
are  derived. 

PROCEDURE 

Each  of  the  military  services  has  issued  policy  directives  and  guidance  relating 
to  the  preparation  of  a  Statement  of  Need.  DoD  Instruction  5000.2  delineates  the  format 
for  an  MNS.  This  format  does  not  differ  appreciably  from  the  formats  used  for  less-than- 
major  new  starts,  thus  the  following  guidance  will  be  discussed  in  relation  to  the  MNS. 

The  SON  is  issued  prior  to  Concept  Exploration.  When  the  Concept  Exploration 
Phase  is  not  conducted,  the  SON  should  be  issued  prior  to  initiation  of  work.  In  addition, 
the  validity  of  the  SON  can  be  reevaluated  prior  to  the  initiation  of  Dem/Val,  FSD,  and 
Production. 


REVIEWING  A  STATEMENT  OF  NEED 


REQUIREMENT  #1.1 


GUIDANCE 


it  is  almost  as  important  to  ensure  what  should  not  be  put  into  an  SON  as  what 
should  be  put  into  an  SON.  From  a  diagnostic  point  of  view,  initially  there  are  no 
diagnostic  requirements  per  se,  only  requirements  which  reflect  a  threat  and  mission  and 
operational  needs,  plus  certain  constraints  put  on  the  weapon  system,  such  as  resource 
limitations.  At  the  initiation  of  a  weapon  system  development,  it  is  important  that  the 
Government  Program  Manager  and  his  contractor  not  be  limited  by  establishing  premature 
diagnostic  requirements,  such  as  a  certain  percent  fault  detection/fault  isolation  to  a  given 
unit,  or  an  MTTR.  Rather,  the  contractor  should  be  given  the  flexibility  to  derive  the 
diagnostic  requirements  from  mission  needs,  such  as  sortie  rates,  mobility  requirements, 
and  the  mission  scenario. 

The  Contractor  Program  Manager  should  examine  the  SON  to  make  sure  he 
and  the  Government  Program  Manager  have  a  common  understanding  of  its  content.  If 
the  SON  contains  needs,  which  the  Contractor  Program  manager  believes  unduly 
constrained  the  efficient  and  effective  development  and  deployment  of  the  diagnostic 
capability,  he  should  inform  the  Government  Program  Manager  so  that  he  may  take  the 
action  which  he  feels  necessary.  As  the  weapon  system  development  progresses  the 
Contractor  Program  Manager  should  periodically  check  the  SON  to  assure  compliance 
with  the  spirit  of  this  document  and  to  make  sure  all  subsequent  proposals  reflect  this 
concern. 
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REVIEWING  A  STATEMENT  OF  NEED 


REQUIREMENT  #1.1 


CHECKLIST 

rf  Does  the  SON  adequately  address  mission  and  threat 
and  refrain  from  prematurely  including  diagnostic 
requirements? 

O'  Have  you  informed  the  Government  Program 
Managers  of  any  potential  deficiencies 
in  the  SON? 
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RESPONDING  TO  AN  RFP,  SOW,  SPECIFICATION  REQUIREMENT  #1.2 


WEAPON 
SYSTEM 
ACQ.  PHASE 

OPER. 

REQMTS. 

CONCEPT 

EXPLOR. 

DEM/ 

VAL 

FSD 

PROD/ 

DEPL. 

WEAPON 

SYSTEM 

ACTIVITY 

A A  A  A 

CONTRACT  AWARD 

DIAGNOSTIC 

ACTIVITIES 

A  A  A  A 

DIAGNOSTIC  INPUTS  TO  PROPOSAL/SPEC. 

DIAGNOSTIC  ACTIVITY 

Clear,  concise,  and  feasible  provisions  must  be  inserted  into  the  Request  for 
Proposal  (RFP),  the  Statement  of  Work  (SOW),  and  the  System  Specification,  as  a  means 
for  assuring  that  the  contractor  and  his  subcontractors  have  a  clear  understanding  of  what 
is  required  of  the  diagnostic  capability. 

The  Contractor  Program  Manager's  job  is  to  respond  to  the  Government 
Program  Manager’s  requirements  with  feasible  and  innovative  diagnostic  alternatives. 

PROCEDURE 


One  of  the  initial  tasks  which  must  be  undertaken  by  the  Government  Program 
Manager,  at  the  beginning  of  each  acquisition  phase,  is  the  development  of  the  Request 
for  Proposal,  which  will  subsequently  lead  to  a  contractual  document. 

Often  a  draft  RFP  will  be  distributed  by  the  Government  Program  Manager  to  all 
interested  bidders  as  a  means  for  assuring  that  the  RFP  will  produce  the  most  effective 
and  efficient  product.  It  is  the  job  of  the  Contractor  Program  Managers  to  work  directly 
with  the  Government  Program  Managers  prior  to  the  final  issuance  of  an  RFP  to  assure 
that  the  requirements  can  be  met  in  a  cost-effective  manner.  Once  the  RFP  has  been 
formally  issued,  it  is  the  Contractor  Program  Manager’s  job  to  respond  to  the  provisions 
contained  in  the  RFP  within  the  bounds  of  a  competitive  environment. 

For  the  Concept  Exploration  Phase,  normally  the  RFP  contains  a  Statement  of 
Work  without  an  associated  weapon  system  specification.  The  specification  is  usually 
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invoked  no  sooner  than  the  Demonstration  and  Validation  Phase.  It  is  normally  written  by 
the  contractor  as  a  data  deliverable,  with  final  review  by  the  Government  Program 
Manager.  The  requirements  for  this  diagnostic  capability  must  appear  In  a  variety  of 
places  throughout  these  documents  to  assure  the  acquisition  of  a  satisfactory  diagnostic 
capability.  For  the  Concept  Exploration  Phase,  these  requirements  are  general  in  nature 
and  allow  the  maximum  flexibility  for  the  contractor  to  do  his  job.  As  the  weapon  system 
design  proceeds,  these  requirements  become  more  and  more  specific.  The  thrust  and 
content  of  the  provisions  contained  in  these  documents  varies,  depending  on  the 
acquisition  strategy  developed  by  the  Government  Program  Manager,  the  phase  in  which 
these  documents  are  invoked,  and  the  size  and  complexity  of  the  weapon  system. 
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GUIDANCE 


The  Contractor  Program  Manager  is  faced  wu*  somewhat  of  a  dilemma  of 
satisfying  the  Government  Program  Manager’s  diagnostic  requirements,  while  placing  his 
corporation  in  a  realistic  competitive  environment.  When  working  with  the  Government 
Program  Manager  prior  to  issuing  a  formal  RFP,  the  Contractor  Program  Manager  should 
stress  the  need  for  clear,  concise,  and  feasible  diagnostic  requirements,  which  promote 
flexibility  in  the  meeting  of  these  requirements.  Once  the  RFP  is  issued,  the  Contractor 
Program  Manager’s  response  must  not  only  be  directed  at  satisfying  the  diagnostic 
requirements,  but  should  permit  the  utilization  of  the  innovative  diagnostic  technology  and 
techniques  to  satisfy  these  requirements.  Integration  of  diagnostic  elements  is  the  central 
issue.  Employment  of  innovative  diagnostic  technology  can  result  in  cost  savings,  not  only 
after  deployment  but  during  design  and  manufacture.  Thus  it  is  possible  to  satisfy 
diagnostic  requirements  utilizing  innovative  techniques  and  still  be  competitive. 

RESPONDING  TO  AN  RFP: 

Diagnostics  impacts  a  number  of  sections  within  an  RFP,  as  shown  in  the 
following  paragraphs. 

Special  Contract  Requirements  (Section  H)  -  Contractor  incentives  and  warranties  are 
contained  in  this  section  of  the  RFP.  The  type  and  content  of  these  incentives  and 
warranties  are  almost  limitless,  depending  on  the  innovation  of  the  RFP  writer.  The 
Defense  System  Management  College  has  published  a  warranty  handbook,  which  is  a 
reference  guide  for  use  by  DoD  managers  in  developing,  applying  and  administering 
warranties.  This  guide  contains: 

o  Warranty  law  and  DoD  Policy 
o  Warranty  concepts  and  issues 
o  Warranty  selection  and  structure 
o  Warranty  development 
o  Warranty  administration 
o  Warranty  cost  benefit  analysis 
o  Case  examples  of  warranties. 

Several  of  these  warranties  are  applicable  to  the  fault  detection,  fault  isolation 
process.  These  deal  with  reliability  improvements,  MTBF  guaranties,  availability 
guaranties,  and  logistic  support  costs  guaranties.  Copies  of  this  document  can  be 
obtained  from  the  Defense  System  Management  College,  who  Is  the  controlling  agency  for 
this  handbook. 

Instructions  to  Offerors  (Section  L)  -  The  contractor’s  response  to  the 
Instructions  to  Offerors  Section  of  the  RFP  is  particularly  important  because  it  addresses 
the  contractor's  understanding  of  the  integrated  diagnostics  process.  Thus  the  proposal 
should  address  the  management  and  technical  approach  relative  to  this  process  and  the 
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meeting  of  the  diagnostic  requirements.  The  proposal  should  emphasize  that  the 
contractor  understands  that  integrated  diagnostics  interfaces  with  logistics,  reliability, 
maintainability,  testability,  human  engineering,  and  safety  requirements.  The  proposal  will 
be  judged  on  how  well  this  integration  is  planned,  organized,  directed,  and  controlled  and 
how  advanced  technology  will  facilitate  this  integration. 

DoD’s  support  of  the  Computer-Aided  Acquisition  and  Logistics  Support  (CALS) 
Program  report  will  be  addressed  in  this  section.  One  of  the  major  parts  of  this  program 
focuses  on  the  automation  of  the  diagnostic  design  process  as  a  means  for  providing  a 
more  efficient  and  effective  design  process.  Usually,  the  government  will  not  dictate  the 
use  of  design  tools,  but  rather  will  encourage  their  use  through  various  incentives.  As  a 
minimum,  the  contractor’s  proposal  should  address  the  following  issues: 

o  A  discussion  of  design  aids  which  will  facilitate  the  design  and  integration  of 
the  diagnostic  capability  into  the  system  engineering  process 

o  The  development  and  use  of  a  diagnostic  data  base  which  supports  the 
application  of  these  tools 

o  Identification  of  how  automation  will  reduce  risk  in  the  design  of  the 
diagnostic  capability 

o  Means  for  providing  the  government  with  appropriate  documentation  for 
understanding  and  validating  the  output  of  the  automation  process 

Additional  information  on  the  implementation  of  CALS  is  contained  in  the  CALS 
implementation  Guide,  which  will  be  issued  as  a  military  handbook. 

Evaluation  Factors  for  Award  (Section  M)  -  This  section  will  be  written  to  assure  that 
the  proposal  writer  understands  that  integrated  diagnostics  and  diagnostic  requirements 
have  a  significant  impact  on  the  selection  of  a  contractor.  The  evaluation  factors  will  most 
likely  reflect  the  diagnostic  content  of  the  instructions  to  Offerors  (Section  L)  from  both 
technical  and  management  points  of  view.  Thus  the  contractor’s  proposal  should  stress 
that  testability  and  integrated  diagnostics  are  part  of  the  system  engineering  process  and 
advanced  technology  will  be  applied  in  solving  this  problem. 

The  contractor  should  emphasize  that  a  single  person  will  be  responsible  for 
managing  the  development  of  the  entire  diagnostic  capability.  This  person  should  have 
systems  engineering  experience  because  he  is  required  to  assure  integration  of  the 
design  of  the  diagnostic  capability,  which  cuts  across  a  multitude  of  design  and 
supportability  functions.  Reliability,  maintainability,  testability,  human  engineering, 
logistics,  etc.  along  with  system,  subsystem,  and  component  diagnostics  are  all  included. 
In  addition,  the  proposal  should  stress: 
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o  The  amount  and  type  of  specialized  education  and  training  given  to  both 
contractor  program  managers  and  designers  which  relate  to  testability  and 
integrated  diagnostics  process 

o  The  independent  research  and  development  conducted  by  the  contractor 
which  relates  to  testability  and  diagnostic  design  tool  development  and 
demonstrations  of  the  integration  of  diagnostic  elements 

o  Methods  and  scheduling  to  assure  the  concurrent  delivery  and  evaluation  of 
the  entire  diagnostic  capability  with  the  prime  system  itself 

o  How  the  diagnostics  for  both  GFE  and  CFE  will  be  addressed  by  the 
contractor  to  assure  that  overall  system  diagnostic  requirements  are  met 

o  The  quality  of  the  diagnostic  maturation  program 

Responding  to  a  Statement  of  Work 

The  character  of  the  Statement  of  Work  will  vary,  depending  on  which  weapon 
system  acquisition  phase  is  being  addressed.  The  responses  to  these  Statements  of 
Work  should  address  t:  ie  factors  described  under  one  or  more  of  these  four  phases  (i.e., 
Concept  Explorat*  \n,  Demonstration  and  Validation,  Full-Scale  Development,  and 
Production).  The  principal  tasks  that  will  be  addressed  throughout  the  development  of  a 
weapon  system  are: 

1 .  An  engineering  analysis  (including  gathering  of  field  data)  from  a  previously 
fielded  weapon  system(s)  to  determine  diagnostic  capability  performance 
deficiencies  experienced 

2.  Identification  of  specific  risk  areas  which  require  design  attention 

3.  A  requirement  for  preparation  and  implementation  of  a  Diagnostic  Capability 
Maturation  Plan,  including  assets  required,  activities  required,  and  data 
collection 

4.  Thorough  analysis  of  the  design  of  the  embedded  diagnostics  to  be 
completed  by  CDR 

5.  Design  analysis  and  specification  of  the  external  diagnostic  capability, 
including  overlap,  by  CDR 

6.  A  requirement  for  demonstration  of  the  diagnostic  capability,  including  a 
thorough,  statistically  valid  sample  in  selected  areas  of  the  system. 
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The  following  are  sample  Statements  of  Work  which  may  be  included  in  the 
SOW  Requirements  section.  When  responding  to  an  RFP  it  is  important  that  the 
Contractor  Program  Manager  address  how  he  plans  to  accomplish  these  tasks.  Even  if 
these  tasks  are  not  called  out  in  the  SOW,  it  may  still  be  crucial  for  these  items  to  be 
addressed.  In  such  a  case,  the  initiative  shown  by  the  contractor  could  be  the  deciding 
factor  in  the  government  source  selection  process. 
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CONCEPT  EXPLORATION  PHASE 

Diagnostic  Approach 

1 .  Establish  overall  diagnostic  design  objectives,  strategies,  goals,  thresholds,  and 
constraints  which  support  mission  requirements  and  operational  constraints  in  support  of 
the  logistic  support  analysis  process  of  MIL-STD-1 388-1  and  the  system  engineering 
process  of  MIL-STD-499.  These  include: 

a.  Translation  of  weapon  system  mission  and  performance  requirements  into 
diagnostic  requirements  for  each  level  of  maintenance  which  support  the 
mission  scenario. 

b.  Establishment  of  requirements  which  allow  for  diagnostic  growth  as  design 
proceeds  through  the  weapon  system  acquisition  phases. 

c.  Identification  of  diagnostics-related  constraints  driven  by  operational 
constraints  of  the  system. 

d.  Identification  of  technology  advancements  which  can  be  exploited  in  system 
development  and  diagnostic  element  development  and  which  have  the 
potential  for  increasing  diagnostic  effectiveness;  reducing  the  requirement  for 
maintenance;  reducing  test  equipment,  technical  manuals  and  manpower, 
and  skill-level  requirements;  reducing  diagnostic  costs;  or  enhancing  system 
availability. 

e.  Identification  of  existing  and  planned  diagnostic  resources  (e.  g.,  family  of 
testers,  maintenance  aids),  which  have  potential  benefits.  Identification  of 
resource  limitations. 

f.  Identification  of  diagnostics  problems  on  similar  systems  which  should  be 
avoided. 

2.  Define  what  constitutes  a  system  failure  and  establishing  deferred  maintenance, 
performance  and  safety  monitoring,  embedded  diagnostic  and  external  diagnostic 
objectives  for  the  new  system  at  the  system  and  subsystem  levels.  Identifying  the  risks 
and  uncertainties  involved  in  achieving  the  objectives  established. 

3.  Establish  BIT,  test  equipment,  technical  information,  and  maintenance  manpower  and 
skill-level  constraints  for  inclusion  in  System  Specifications  or  other  requirements 
documents.  These  constraints  shall  include  both  quantitative  and  qualitative  factors. 

4.  Evaluate  alternative  diagnostic  concepts  to  include  varying  degrees  of  BIT,  manual  and 
automatic  testing,  technical  information  format  and  delivery  systems,  personnel  and 
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training,  along  with  deferred,  preventive,  and  scheduled  maintenance  concepts,  and 
identify  the  selected  concept.  The  evaluation  includes: 

a.  A  determination  of  the  sensitivity  of  system  mission  performance  and 
readiness  parameters  to  variations  in  key  diagnostic  element  parameters 

b.  A  determination  of  the  sensitivity  of  life  cycle  costs  to  variations  in  diagnostic 
element  parameters 

c.  An  estimation  of  the  manpower  and  personnel  implications  of  alternative 
diagnostic  concepts  in  terms  of  direct  maintenance  manhours  per  operating 
hour,  job  classification,  skill  levels,  and  experience  required  at  each  level  of 
maintenance 

d.  An  estimation  of  the  risk  associated  with  each  concept. 

Diagnostic  Program  Planning 

Develop  a  Diagnostic  Capability  Program  Plan  which  describes  how  the 
program  will  be  conducted.  This  program  plan  may  be  included  as  part  of  the  System 
Engineering  Management  Plan  (SEMP).  The  plan  should  describe  the  time  phasing  of 
each  task  including  the  contractual  requirements  and  its  relationship  to  other  tasks. 

The  plan  should  also  address  the  following: 

a.  Identify  a  single  organizational  element  within  the  performing  activity  which 
has  overall  responsibility  and  authority  for  implementation  of  the  program. 
Establish  analyses  and  data  interfaces  among  the  organizational  elements 
responsible  for  each  of  the  elements  of  the  diagnostic  capability  and  other, 
related  elements. 

b.  Develop  a  process  by  which  diagnostic  requirements  are  integrated  with 
other  design  requirements  and  disseminated  to  design  personnel  and 
subcontractors.  Establish  controls  for  assuring  that  each  subcontractor’s 
diagnostic  practices  are  consistent  with  overall  system  or  equipment 
requirements. 

c.  Identify  diagnostic  design  guides,  analysis  models  and  procedures  to  be 
imposed  upon  the  design  process.  Plan  for  the  review,  verification,  and 
utilization  of  diagnostic  data  submissions.  Explain  how  computer-aided 
design  tools  will  be  utilized  in  this  process. 
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L 


J 


Diagnostic  Program  Reviews 

Describe  the  conduct  of  the  diagnostic  portion  of  the  System  Requirements 
Review  (SRR),  including  how  these  reviews  will  interrelate  with  reliability,  maintainability, 
human  engineering,  and  logistic  support  reviews. 


> 
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DEMONSTRATION  AND  VALIDATION  PHASE 
Diagnostic  Specification  Development 

1 .  Perform  detailed  comparability  and  design  analysis  and  risk  reduction  efforts  necessary 
to  develop  a  specification  provision  which  allocates  testability/diagnostic  requirements. 
These  are  used  for  fault  detection/isolation,  repair  verification,  performance  or  safety 
monitoring,  and  damage  assessment.  This  enables  the  weapon  system  to  meet 
maintenance  and  operational  goals  with  a  minimum  of  unnecessary  removals.  Diagnostic 
capabilities  should  be  selected  from  design  techniques  (including  built-in  test,  fault 
tolerance,  status  monitoring,  partitioning,  test  points);  external  hardware  (e.  g.,  automatic 
and  manual  test  equipment  and  maintenance  aids);  technical  information  (e.  g.,  technical 
manuals,  information  systems,  and  operator  displays);  and  training  manuals  (e.  g.,  formal 
schooling,  on-the-job  training).  The  capabilities  selected  may  be  designed  into  the  system 
as  part  of  the  system  and/or  provided  separately  to  maintenance  personnel,  as  required, 
to  meet  mission  and  level-of-repair  objectives. 

2.  Based  on  the  results  of  the  analyses  and  risk  reduction  efforts,  the  diagnostic 
capabilities  to  be  provided  with  the  system  at  each  level  of  maintenance  should  be 
specified.  This  includes: 

a.  Mode  of  operation  (e.  g.,  status  monitoring)  in  areas  where  there  is 
diagnostic  ambiguity  or  overlap 

b.  Operational  test  strategies,  fault  tolerance,  prognostics,  and  fault  model 
assumptions 

c.  Performance  in  terms  of  mean  time  to  diagnose,  fault  coverage,  false 
alarms,  and  false  removals 

d.  Physical  and  functional  equipment  partitioning  requirements 

e.  Physical  (weight,  volume)  and  functional  (%  memory)  limitations 

f.  Diagnostic  capability  interface  requirements 

g.  Options  for  augmenting  government-furnished  equipment  (GFE)  diagnostic 
capabilities 

h.  Reliability  of  the  BIT  and  external  diagnostic  hardware. 
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Detailed  Diagnostic  Comparison  Analysis 

1 .  As  part  of  the  development  of  the  diagnostic  specification  provisions,  perform  a 
comparison  analysis,  using  the  baseline  fielded  system  at  each  level  of  field  maintenance. 
This  should  include  an  analysis  of  the  causes  of  diagnostic  times,  undetected  faults,  "false 
alarms,"  and  "false  removals"  which  are  judged  to  be  excessive.  The  sources  of  these 
causes  need  to  be  identified  and  the  improvements  of  a  new,  proposed  system  diagnostic 
design  must  be  delineated.  As  a  minimum,  this  analysis  characterizes  whether  the  causes 
of  diagnostic  problems  are  inherent  to  the  design  (e.  g.,  reliability,  maintainability, 
partitioning,  connectors,  etc.),  due  to  maintenance  procedures,  lack  of  "vertical"  testability 
(e.  g.,  cone  of  tolerance,  compatibility  between  levels  of  maintenance),  or  transients. 

2.  Provide  quantitative  assessments  of  diagnostic  capabilities  identifying  current 
capabilities,  extrapolations  to  proposed  capabilities,  and  the  engineering  analysis  that  is 
the  basis  for  the  extrapolation.  Overlaps  or  ambiguities  in  diagnostic  capabilities  used  for 
maintenance  of  fielded  systems,  must  be  identified  and  addressed  for  the  proposed 
system.  When  deficiencies  in  the  GFE  preclude  meeting  (he  diagnostic  requirements, 
alternatives  need  to  be  addressed.  Weight  and  volume  of  the  major  external  test 
equipment,  type  and  extent  of  technical  information,  maintenance  skill  levels  and  training 
requirements  for  currently  fielded  systems  should  be  identified  and  estimates  of  these 
quantities  for  the  proposed  diagnostic  capability  and  explanation  of  the  basis  for  this 
estimate  need  to  be  provided. 

Diagnostic  Risk  Reduction 

1 .  As  part  of  the  design,  prototype,  test,  and  demonstration  activities  proposed  (the  basis 
of  the  proposal  shall  be  risk  areas  identified  in  Concept  Exploration),  determine  the 
feasibility  of  achieving  diagnostic  capability  performance  improvements. 

Diagnostic  Maturation  Planning 

1.  Develop  a  plan  for  next  phase  activities  in  the  areas  of  analysis,  growth,  and 
demonstration  of  the  entire  diagnostic  capability  (hardware  and  software).  The  plan 
should  also  include  the  resources  required  for  maturation  activities  (e.  g.f  prime  hardware, 
laboratory  facilities). 

2.  Design  a  diagnostic  data  system  which  extends  from  Demonstration  and  Validation 
through  Full-Scale  Development,  Production,  and  Deployment.  The  data  system  should 
be  designed  so  that  the  performance  of  the  diagnostic  capability  can  be  ascertained  at  any 
point  during  the  acquisition,  production,  and  deployment  of  the  weapon  system,  and  be 
compatible  with  the  established  DoD  data  system,  which  will  be  employed  after  the 
maintenance  of  the  weapon  system  becomes  the  responsibility  of  the  government. 
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Testability,  Preliminary  Design 

1.  Apply  testability  design  criteria  to  the  design  of  selected  high-risk  items,  in  accordance 
with  MIL-STD-2165,  Task  202.2.1.  The  testability  design  criteria  to  be  considered  should 
include  selective  implementation  of  system-level  diagnostic  strategies,  partitioning  to 
enhance  fault  isolation,  initialization  of  circuitry  under  test  control,  module  interface  for  test 
access  and  control,  circuit  controllability  and  observability,  parts  selection,  test  point 
placement,  and  BIT  fault  detection  approaches. 

Diagnostic  Capability  Planning 

1 .  Develop  a  Diagnostic  Capability  Program  Plan  which  describes  how  the  program  will  be 
conducted.  This  program  plan  may  be  included  as  part  of  the  System  Engineering 
Management  Plan.  The  plan  should  describe  the  time  phasing  of  each  task  included  in 
the  contractual  requirements  and  its  relationship  to  other  tasks.  Diagnostic  issues  which 
relate  to  reliability,  maintainability,  logistics,  human  engineering,  safety,  etc.,  should  be 
addressed  in  those  individual  plans. 

The  plan  should  address,  but  not  be  limited  to,  the  following  requirements: 

a.  Identify  a  single  organizational  element  within  the  performing  activity  which 
has  overall  responsibility  and  authority  for  implementation  of  the  program. 
Establish  analyses  and  data  interfaces  among  the  organizational  elements 
responsible  for  each  of  the  elements  of  the  diagnostic  capability  and  other 
related  elements. 

b.  Develop  a  process  by  which  diagnostic  requirements  are  integrated  with 
other  design  requirements  and  disseminated  to  design  personnel  and 
subcontractors.  Establish  controls  for  assuring  that  each  subcontractor’s 
diagnostic  practices  are  consistent  with  overall  system  or  equipment 
requirements. 

c.  Identify  diagnostic  design  guides,  analysis  models  and  procedures  to  be 
imposed  on  the  design  process.  Plan  for  the  review,  verification,  and 
utilization  of  diagnostic  data  submissions.  Explain  how  computer-aided 
design  tools  will  be  utilized  in  this  process. 

Diagnostic  Program  Reviews 

1 .  Describe  the  conduct  of  the  diagnostic  portion  of  the  System  Design  Review  (SDR) 
including  how  these  reviews  relate  to  reliability,  maintainability,  human  engineering,  and 
logistic  support  reviews. 
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FULL-SCALE  DEVELOPMENT  PHASE 
Design  of  the  Diagnostic  Capability 

1 .  Describe  how  the  embedded  diagnostics  and  testability  features  will  be  incorporated  in 
the  system  and  how  the  external  diagnostics  capabilities  which  will  satisfy  the  diagnostic 
capability  performance  requirements  will  be  provided. 

Diagnostic  Design  Analysis 

1 .  Implement  a  structured  design  analysis  process  to  assess  in  detail  the  ability  of  the 
diagnostic  capability  design  to  meet  the  system  diagnostic  performance  specification  (e.g., 
fault  coverage,  mean  time  to  diagnose,  false  removal,  etc.);  analyzing  the  inherent 
testability  of  the  preliminary  design;  identifying  the  areas  where  the  primary  means  of 
diagnostics  may  lead  to  an  ambiguous  result  and  ways  the  ambiguity  will  be  resolved; 
identifying  areas  where  there  is  a  redundant  (overlapping)  diagnostic  capability  planned  to 
be  provided;  and  verifying  that  the  detailed  design  of  diagnostics  is  in  accordance  with  the 
functional  allocation  established  during  the  previous  program  phase.  The  analytical  task 
should  include: 

a.  Design  Analysis  of  Diagnostics  Built  In  the  System 

Conduct  a  structured  analysis  of  system  design  implementation  to  identify  the 
functional  areas  in  which  diagnostics  capabilities  allocated  in  the  previous  phase 
to  be  built  into  the  system  provide  an  unambiguous  capability  to  detect  or  isolate  a 
fault  to  the  appropriate  replaceable  unit  at  each  level  of  maintenance. 

b.  Assessment  of  External  Diagnostics 

Deliver  at  the  Critical  Design  Review  detailed  requirements  definition  for  external 
test  equipment,  troubleshooting  approaches  to  be  included  in  maintenance 
manuals,  maintenance  aids,  and  training  requirements.  These  requirements 
should  each  be  supported  by  a  diagnostic  ambiguity  analysis  to  be  delivered  at  the 
same  time,  which  describes  the  degree  to  which  diagnostic  ambiguities  are 
reduced  and  the  areas  where  there  is  redundancy  (overlap)  of  diagnostic 
capabilities. 

Diagnostic  Maturation  Program 

1.  Establish  and  maintain  a  diagnostic  performance  data  collection  system  and 
conducting  diagnostic  performance  verification  tests  and  demonstrations  to  evaluate  the 
effectiveness  of  the  diagnostic  design.  The  total  diagnostic  capability,  both  embedded  and 
external,  should  be  evaluated  concurrently  with  the  weapon  system  itself  for 
maintainability  demonstrations  as  well  as  for  test  and  evaluations. 


1-19 


RESPONDING  TO  AN  RFP,  SOW,  SPECIFICATION  REQUIREMENT  #1.2 


2.  Monitor  diagnostic  performance  whenever  the  system  is  operating  and  performing  an 
analysis  to  determine  whether  the  diagnostic  capabilities  are  operating  in  accordance  with 
the  design.  Based  on  the  maturation  results,  taking  corrective  action  in  order  to  meet 
diagnostic  capability  requirements. 

3.  Plan  for  the  transition  of  responsibility  to  the  government  for  the  collection  and  analysis 
of  diagnostics  data.  Field  data  collection  and  analysis  should  be  automated  to  the 
maximum  extent  practical  and  cost  effective;  and  integrated  to  the  maximum  extent 
practical  with  similar  data  collection  requirements  specified  elsewhere. 

Diagnostic  Capability  Program  Planning 

1 .  Develop  and  maintain  a  Diagnostic  Capability  Program  Plan  which  describes  how  the 
program  will  be  conducted.  The  program  plan  may  be  included  as  part  of  the  SEMP.  The 
plan  should  describe  the  time  phasing  of  each  task  included  in  the  contractual 
requirements  and  its  relationship  to  other  tasks.  Diagnostic  issues  which  relate  to 
reliability,  maintainability,  logistics,  human  engineering,  safety,  etc.,  should  be  addressed 
in  those  individual  plans. 

The  plan  should  address,  but  not  be  limited  to,  the  following  requirements: 

a.  Identify  a  single  organizational  element  within  the  performing  activity  which 
has  overall  responsibility  and  authority  for  implementation  of  the  program. 
Establish  analyses  and  data  interfaces  among  the  organizational  elements 
responsible  for  each  of  the  elements  of  the  diagnostic  capability  and  other 
related  elements. 

b.  Develop  a  process  by  which  diagnostic  requirements  are  integrated  with 
other  design  requirements  and  disseminated  to  design  personnel  and 
subcontractors.  Establish  controls  for  assuring  that  each  subcontractor’s 
diagnostic  practices  are  consistent  with  overall  system  or  equipment 
requirements. 

c.  Identify  diagnostic  design  guides,  analysis  models  and  procedures  to  be 
imposed  upon  the  design  process.  Plan  for  the  review,  verification,  and 
utilization  of  diagnostic  data  submissions.  Explain  how  computer-aided 
design  tools  will  be  utilized  in  this  process. 

Diagnostic  Program  Reviews 

1 .  Describe  the  conduct  of  the  diagnostic  portion  of  the  formal  reviews  (e.  g.,  PDR,  CDR) 
which  are  conducted  during  FSD,  including  how  these  reviews  relate  to  reliability, 
maintainability,  human  engineering,  and  logistic  support  reviews. 
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PRODUCTION  PHASE 


DIAGNOSTIC  MATURATION 

1 .  Mature  the  diagnostic  capability  in  accordance  with  the  established  Maturation  Plan  to 
assure  that  required  improvements  are  made  toward  satisfying  the  updated  Diagnostics 
Performance  Specification  at  each  maintenance  level.  This  includes: 

a.  Maintaining  and  utilizing  the  diagnostic  data  system  to  measure 
performance  of  the  diagnostic  capability  and  take  required  corrective  action, 
in  accordance  with  the  incentive  and  warranty  provisions 

b.  Planning  and  transitioning  of  the  data  analyses  and  system  to  the 
government 

c.  Demonstrating  that  the  diagnostic  capability  satisfies  the  diagnostic 
requirements. 
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PREPARING  A  SPECIFICATION: 

Preparation  of  the  diagnostic  portion  of  a  weapon  system  specification  is  a  job 
which  necessitates  a  full  understanding  of  the  design  and  fielding  of  the  diagnostic 
capability.  When  preparing  this  specification,  the  Contractor  Program  Manager  must 
recognize  the  intricacies  of  this  job  to  ensure  that  the  specifications  utilized  to  acquire  a 
weapon  system  dearly  define  the  diagnostic  requirements. 

What  Is  a  Failure? 

An  initial  requirement  in  the  specification  is  to  establish  the  definition  of  a  failure 
at  system,  subsystem,  and  unit  levels.  This  requirement  is  essential  in  demonstrating 
graceful  degradation  through  the  use  of  fault-tolerant  design,  reconfigurability, 
redundancy,  and  performance  monitoring.  A  failure  may  be  defined  as  causing  the 
mission  and  performance  requirements  of  the  prime  system  to  be  compromised. 

What  Means  are  Available  to  Perform  Diagnostics? 

Too  often  the  word  "diagnostic"  is  used  interchangeably  with  "test."  The 
specification  must  recognize  the  different  types  of  diagnostics,  which  include: 

o  Sensory  observations  (e.g.,  the  display  isn’t  onl) 

o  Symptomatic  (e.g.,  usually  this  means  the  power  supply  isn’t  working I) 

o  Test  (e.g.,  an  output  voltage  measures  zero!) 

Means  through  which  systems’  diagnostics  can  be  addressed  include: 

o  Automatic  testing  (i.e.,  embedded  or  external) 

o  Manual  troubleshooting,  utilizing  technical  manuals,  troubleshooting 
procedures,  manual  test  equipment 

o  Operator  and  maintenance  technicians’  observations  and  various  forms  of 
performance  monitoring 

o  A  combination  of  the  above. 

What  Terms  Can  be  Used  In  Specifying  Diagnostic  Requirements? 

As  indicated  previously,  various  terms  may  be  used  to  specify  diagnostic 
requirements.  A  preferred  set  is  contained  in  an  RADC  report,  "A  Rationale  and  Approach 
for  Defining  and  Structuring  Testability  Requirements,"  (RADC-TR-85-150),  August  1985. 
The  set  includes  the  following: 
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Fraction  of  Faults  Detected  (FFD) 

FFD  can  be  defined  as  that  fraction  of  failures  which  occur  over  operating 
time  which  can  be  correctly  identified  through  direct  observation  or  other  specified  means 
by  an  operator  and/or  other  specified  personnel  under  a  given  set  of  conditions.  The 
quantitative  definition  of  FFD  is: 

FFD  -  Fd 


Where: 

Fa  -  Number  of  actual  failures  (faults)  which  (will) 
occur  over  operating  time,  T. 

Fp  *  Number  of  actual  failures  correctly  identified 
through  direct  observation  and  other  specified 
means  by  an  operator  and/or  other  specified 
personnel  under  a  given  set(s)  of  conditions. 

In  specifying  FFD,  all  the  various  means  which  can  be  used  to  detect  faults  must 
be  taken  into  consideration.  The  requirement  for  FFD  should  be  stringent  enough  to 
exclude  the  application  of  the  types  of  detection  means  which  are 
unsatisfactory/unacceptable  for  the  system  needs/objectives/philosophies,  but  flexible 
enough  to  allow  the  contractor  to  tailor  his  design  cost  effectively.  In  general,  the  specific 
nature  and  mix  of  the  means  to  be  employed  to  achieve  a  given  minimum  FFD  should  be 
dependent  on  results  of  an  analysis  of  each  such  alternative  and  its  cost  and  performance 
effectiveness,  in  conjunction  with  other  system/equipment  design  factors  and 
requirements.  The  contractor  should  be  tasked  to  perform  such  analyses  and  provide 
results/recommendations  to  the  procuring  activity  based  on  these  factors. 

The  FFD  specification  parameter  must  be  specifically  defined  to  take  into 
account  frequency  of  failure  (failure  rates)  of  the  components  making  up  the  system.  It  is 
only  in  this  way  that  FFD  will  be  representative  of  what  occurs  during  operational  life. 

In  specifying  FFD,  care  must  be  taken  to  define  that  set  of  detection  conditions 
which  are  acceptable:  for  example,  who  can  perform  the  detection  function;  what  are  the 
acceptable  means  through  which  detection  can  be  performed;  during  which  equipment 
status  modes  can  detection  be  performed  (operation,  pre-  or  post-mission  checks,  etc.); 
and  whether  or  not  a  failure  must  be  detected  within  a  certain  period  of  time?  It  should 
also  be  noted  that  the  fraction  of  faults  detected  will  have  a  significant  effect  on  the 
reliability  of  the  fault-tolerant  system  if  diagnostics  are  an  essential  part  of  the  system’s 
failure/recovery  mechanism. 
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Fraction  of  Faults  Isolated  (FF1) 

FFI  can  be  defined  as  that  fraction  of  failures  which  occur  over  operating  time 
which  can  be  correctly  isolated  to  x  units,  or  fewer,  at  a  given  maintenance  echelon 
through  use  of  specified  means,  by  a  maintenance  technician  or  other  specified  personnel. 
The  quantitative  definition  for  FFI  is: 

FFI  -  F, 

^A 

Where: 

F^  >  Number  of  actual  failures  (faults)  which  (will) 
occur  over  operating  time  T. 

Fj  >  Number  of  actual  failures  (faults)  which  (will) 
occur  over  operating  time  T  that  can  be  correctly 
isolated  to  x  units,  or  fewer,  at  a  given  maintenance 
echelon  through  use  of  specified  diagnostic  scheme(s)/ 
procedure(s)  (or  a  defined  set  of  such),  by  a 
maintenance  technician  or  other  specified  personnel. 

In  specifying  FFI,  all  the  various  generic  means  acceptable  in  general  for  the 
mission/operational/maintenance  environment  which  can  be  used  to  isolate  faults  must  be 
taken  into  consideration.  The  requirement  for  FFI  should  be  stringent  enough  to  exclude 
the  application  of  isolation  means  which  are  known  in  general  to  be 
unsatisfactory/unacceptable  to  the  system  needs/maintenance  philosophy/objectives  but 
are  flexible  enough  to  allow  the  contractor  to  tailor  his  design  cost  effectively.  The  specific 
nature  and  mix  of  the  means  to  be  employed  should  be  dependent  on  the  results  of  an 
analysis  task  (levied  on,  and  performed  by,  the  contractor)  of  each  fault  isolation 
alternative,  in  conjunction  with  system/equipment  design  factors,  maintainability 
requirements,  and  support  system  needs.  Generally  speaking,  unless  there  is  clear 
evidence  that  unacceptable  weight,  volume,  or  cost  penalties  would  accrue,  primary 
diagnostic  means  based  on:  (1)  signal  tracing  and  analyses  through  the  use  of  schematics 
and  test  equipment,  and  (2)  repetitive  item  remove/replacement/performance  check 
actions  should  be  avoided. 

In  specifying  FFI  care  must  be  taken  to  indicate  the  conditions  under  which 
isolation  must  take  place: 

o  Where  it  takes  place  (i.e.,  Organizational  Level,  Shop  Level) 
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o  What  are  the  acceptable  means  of  isolation  (i.e.,  built-in  test,  external 
testers,  general-purpose  testers,  peculiar  testers,  manual  means,  degree  of 
manual  means) 

o  Who  will  perform  the  isolation  (i.e.,  operator  or  maintenance  technician) 

o  Its  constraints  (i.e.,  prohibition  of  wholesale  removal  of  units,  time  allowable) 

o  Its  second  isolation  tier  requirements  (what  happens  after  isolation  to  proper 
ambiguity  level) 

o  The  time  constraints  levied  by  the  maintainability  requirement. 

The  FFI  parameter  must  be  specifically  defined  to  take  into  account  frequency  of 
failure  (failure  rates)  of  the  components  making  up  the  system.  It  is  only  in  this  way  that 
FFI  will  be  representative  of  what  occurs  during  operational  life. 

False  Alarm  Rate 

A  false  alarm  is  defined  as  an  apparent  indication  of  failure  when,  in  fact,  no 
failure  exists.  The  false  alarm  rate  is  the  number  of  false  alarms  per  unit  of  time. 

Intermittent  faults  can  be  difficult  to  distinguish  from  false  alarms  during 
operational  test  and  in  use.  A  properly  structured  qualification  test,  however,  can  exclude 
the  influence  of  intermittent  faults. 

False  aftirm  rates  are  controllable  through  the  use  of  such  design  techniques 
and  features  as: 

o  Scope  and  magnitude  of  performance  monitoring 

o  Definition  of  test  tolerances 

o  Transient  monitoring  and  control 

o  Multiple-run  decision  logic 

o  Environmental  effects  filtering  and  identification. 

Fraction  of  Erroneous  Fault  Isolation  Results  (FEFI) 

FEFI  is  the  fraction  of  BIT  or  external  tester  isolations  that  identify  the  wrong 
removable  unit  (subunit)  or  group  of  units  (subunits)  as  failed.  FEFI  is  primarily  a  design 
problem  resulting  either  from  test  system  design  error  or  low  sensitivity  thresholds  and 
tolerance  levels  of  system/equipment  components  and/or  signals.  It  can  have  serious 
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consequences  by  creating  confusion  during  fault  isolation  and  by  eroding  maintenance 
technician  confidence  in  the  test  system.  The  quantitative  definition  is: 

FEFI  -  Fe 

Fa 


Where: 

FA  -  Number  of  actual  failures  (faults)  which  (will) 
occur  over  operating  time  T. 

Fe  -  Number  of  actual  failures  (faults)  which  (will) 
occur  over  time  T  that  are  isolated  to  a 
nonfailed  unit  or  group  of  units. 

What  Does  100%  Fault  Detectlon/Fault  Isolation  Mean? 

In  defining  FFD  as  a  contractual  requirement  for  most  programs,  it  is  sometimes 
simpler  to  exclude  those  types  of  direct  detection  means  (for  example,  detection  through 
the  use  of  technicians)  which  would,  in  general,  be  unsatisfactory  to  a  given  mission 
environment  than  to  define  those  that  are  acceptable.  The  fact  that  an  FFD  requirement  is 
imposed  should  not  imply  that  100%  of  all  expected  failures  should  not  be  detectable.  The 
contractor  should  be  tasked  with  the  development  of  cost-effective,  defined  procedures  to 
detect  all  expected  failures.  All  of  these,  however,  need  not  be  direct  means  or  belong  to 
the  type  of  direct  means  which  are  defined  as  satisfactory  for  general  mission  operational 
use,  provided  maintainability  and  other  requirements  can  still  be  met.  Detection  can 
include  direct  or  indirect  indications  to  an  operator,  the  use  of  maintenance  technicians  or 
other  personnel  performing  in  accordance  with  a  series  of  defined  routines,  or  some 
combination  of  these. 

For  FFI  100%  coverage  is  required,  which  simply  states  that  using  a 
combination  of  all  diagnostic  resources,  all  faults  can  be  isolated,  given  an  adequate 
amount  of  time.  Applying  restrictions  in  time  means  that  1 00%  of  all  expected  faults  will  be 
Isolatable.  but  a  certain  fraction  (1-FFI)  may  have  ambiguity  levels  greater  than  the  value 
stated  or  be  Isolatable  through  means  which  are  definable,  but  which  do  not  belong  to  the 
class  of  diagnostic  means  cited  as  being  acceptable  for  general  use  in  the  given  mission 
or  use  environment.  Consideration  must  be  given  as  to  how  and  where  isolation  to  the 
faulty  unit(s)  must  take  place. 

In  summary,  specifications  should  indicate  a  100%  fault  detection/fault  isolation 
coverage  at  each  maintenance  level  (e.g.,  combinations  of  automatic  and  manual 
troubleshooting  means  should  equal  100%).  This  does  not  mean  that  100%  of  faults  can 
be  isolated  to  a  given  unit  within  a  given  time  using  specific  diagnostic  resources. 
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What  is  Diagnostic  Growth? 

Another  aspect  which  is  recommended  to  be  introduced  is  that  of  diagnostic 
growth-similar  in  concept  to  the  already  established  reliability  growth.  This  growth 
requirement  is  especially  important  in  the  maturation  of  the  system.  Figure  2  is  a 
conceptual  version  of  this  growth  process.  Demonstrations  that  these  goals,  or 
requirements,  have  been  achieved  at  various  phases  of  weapon  system  development 
must  be  tailored  to  the  specific  weapon  system  acquisition  strategy.  For  instance,  if  the 
performance  of  an  aircraft  is  to  be  evaluated  at  the  conclusion  of  Dem/Val,  then  the  entire 
diagnostic  capability  for  the  aircraft  should  reach  the  specified  requirement  at  that  point  in 
time.  On  the  other  hand,  if  only  specific  units  (usually  high  risk)  of  a  weapon  system  are 
developed  during  Dem/Val,  then  the  diagnostic  capability  for  only  those  specific  units  may 
be  demonstrated.  The  maturation  of  a  diagnostic  capability  for  the  entire  weapon  system, 
in  most  cases,  will  extend  into  the  Deployment  Phase. 


GOAL  GOAL  GOAL  GOAL 


EXPLORATION  VAL 

FIGURE  2.  DIAGNOSTIC  GROWTH  CONCEPT 


1-27 


RESPONDING  TO  AN  RFP,  SOW.  SPECIFICATION 
What  Does  a  "Diagnostic  Specification"  Contain? 


REQUIREMENT  #1.2 


The  diagnostic  portions  of  the  weapon  system  specification  differ,  depending  on 
the  stage  of  development.  Normally,  these  specifications  take  the  form  oi  a  Preliminary 
System  Specification,  resulting  from  Concept  Exploration;  a  System  Specification,  derived 
from  the  Demonstration  and  Validation  Phase;  and  a  Configuration  Item  Development 
Specification,  which  allocates  requirements  down  to  subsystem  and  item  levels.  (A  more 
complete  definition  of  the  various  types  of  specifications  is  contained  in  MIL-STD-490, 
Specification  Practices.)  The  following  examples  of  diagnostic  portions  of  specifications 
follow  this  form.  The  content  must  be  tailored  to  fit  a  specific  system  requirement. 

Preliminary  System  Specification 

The  Preliminary  System  Specification  is  a  result  of  Concept  Exploration  Phase 
studies,  prior  to  conducting  the  detailed  diagnostic/testability  requirements  analysis  during 
the  Demonstration  and  Validation  Phase. 

Quantitative  testability/diagnostic  parameters  are  not  specified  in  the  Preliminary 
System  Specification.  Rather,  qualitative  system-level  diagnostic/testability  goals  are 
included. 


The  model  paragraphs  below  may  be  included  in  the  Preliminary  System 
Specification  primarily  to  alert  the  performing  activity  that  diagnostic/testability  is 
considered  to  be  an  important  aspect  of  the  design  and  that  quantitative  requirements  will 
be  imposed  in  the  final  System  Specification. 


3JLX  Design  for  Testability 

3.XJC.1  Partitioning.  The  system  shall  be  partitioned  based,  In  part,  npon 
the  ability  to  confidently  Isolate  fa  nits. 

3.XJC.2  Test  Points.  Each  onM  within  the  system  shall  have  sufficient  test 
points  for  the  measurement  or  stimulus  of  internal  circuit  nodes  so  as  to 
achieve  an  Inherently  Ugh  level  of  fault  detection  and  Isolation. 

3.XJL3  Built-In  Test.  Mission  critical  fractions  shall  be  monitored  by 
Built-In  Test  BIT  tolerances  shall  be  set  to  optimize  fruit 
detectlon/febe  alarm  characteristics.  BIT  Indicators  shall  be  designed 
for  maximum  utilization  by  Intended  personnel  (operator/ main talner). 

System  Specification 

Quantitative  testability/diagnostic  requirements  are  derived  from  the  tradeoff 
analysis  during  the  Demonstration  and  Validation  Phase  and  are  incorporated  in  the 
System  Specification.  Requirements  may  be  expressed  in  terms  of  goals  and  thresholds 
rather  than  as  a  single  number.  Requirements  for  diagnostic/testability  in  a  System 
Specification  are  provided  in  the  following  model. 


MODEL/SYSTEM  SPECIFICATION 


3.2.4. X  Dlagnostlc/Testablllty-The _ system  ( Insert  name)  shall  be  designed  for 

testability  and  constructed  to  permit  the  status  of  the  system  and  the  unamblguor - 
location  of  faults  to  be  confidently  determined  and  reported  In  a  timely  fashion. 

3.2.4. X.  1  Partitioning  (Functional  Modularity)  -  System/Subsystems  will  be  partitioned 
Into  Line  Replaceable  Units  (LRU)  based  on  the  function,  minimum  or  optimum  number 
of  Interconnections,  the  ability  to  fault  Isolate  to  the  correct  unit.  LRUs  will  be 
subdivided  Into  next  level  replaceable  Items  (e.g.,  Shop  Replaceable  units,  or  SRU) 
based  on  function,  minimum  or  optimum  number  of  Interconnections,  and  the  ability  of 
personnel  with  the  aid  of  support  equipment,  training,  and  technical  manuals  to  fault 
Isolate. 

3.2.4. X.2  Test  Points  and  Contacts  -  Test  points  and  contacts  shall  be  conveniently 
located  and  have  safe  access  to  signal  nodes  and  shall  be  provided  for  the  measure  or 
Injection  to  significant  parameters  for  the  purpose  of  evaluating  or  troubleshooting  the 
circuit  mechanisms,  the  number  and  choice  of  accessible  nodes  shall  be  sufficient  to 
obtain  the  equipment  fault  detectlon/lsolatlon  requirements  listed  herein. 

3.2.4. X.3  Diagnostic  Capability  -  For  each  level  of  maintenance:  all  diagnostic  resources 
shall  be  Integrated  to  provide  a  consistent  and  complete  diagnostic  capability.  A 
complete  diagnostic  capability  must  Identify  the  diagnostic  resources  that  will  be  used  to 
have  full  FD/FI  coverage.  The  degree  of  diagnostic  automation  shall  be  consistent  with 
the  proposed  personnel  skill  levels  and  maintenance  repair  times. 

3.2.4. X.4  Bullt-ln-Test  -  Built-In  Test  (BIT)  provisions  shall  be  designed  Into  the 

_ system  to  test  system/equipment  and  to  Inform  the  operator  of  the  ability 

of  the  equipment  to  perform  a  particular  mission. 

3.2.4.  X.4.1  On-Line  BIT  Performance  Monitoring  -  The  on-line  BIT  performance 
monitoring  Matures  shall  be  operative  and  shall  provide  valid  performance  Indications 
prior  to  and  during  operation.  The  performance  monitoring  operation  shall  be  automatic 
and  continuous  and  shall: 

1.  Ensure  subsystems  are  operational  and  are  capable  of 
satisfying  their  designated  mission  functions. 

2.  Detect  any  system  failure  or  degradation  which  would  adversely 
affect  the  system’s  ability  to  satisfy  Its  mission  objectives. 

All  BIT  Implementations  of  this  requirement  shall  be  contained  within  the  system  or 
subsystem  hardware  and  will  not  degrade  mission  performance  at  any  time. 
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3.2.4. X.4.1.1  NO-GO  Condition  Dotoctlon  -  Tho  system  on-line  BIT  performance 

monitoring  features  shall  detect  at  least _ percent  of  all  NO-GO  condition 

occurrences  over  the  mission  time.  (As  applied  at  the  weapon  system  level  or  as  applied 
Independently  at  the  subsystem  level.) 

3.2.4. X.4.1.2  False  Alarms  -  The  number  of  false  alarms  shall  not  exceed _ 

percent  of  all  Indicated  NO-GO  condition  occurrencoa  or  alternately  no  more  than 

_ Indicated  NO-GO  condition  occurrences  In  any  24-hour  period  of  system 

operating  time. 

3.2.4. X.4. 1.3  Performance  Monitor  and  Self-Test  Data  -  Performance  monitor  and  self-test 
data  shall  be  transmitted  In  a  manner  such  that  the  transmitted  data  shall  follow  the 
actual  condition  of  the  system,  that  la,  a  malfunction  which  corrects  Itself  shall  change 
the  fault  data  line  accordingly. 

3.2.4 JC.4.2  Off-Line  BIT-  The _ system  BIT  provision  shall  furnish  the  means  for 

an  operator  to  Initiate  BIT  tests  for  purposes  of  determining  and  displaying  the  functional 
status  of  the  systems/subsystems  Including  a  fault  detection  and  Isolation  capability. 
The  Intended  use  of  the  off-line  BIT  tests  la  two-fold:  as  a  System  readiness  test  to 
permit  operating  crew  to  accumulate  status  and  fault  Informatldn  on  an  opportunity  basis 
prior  to  and  during  operations;  and  to  verify  a  fault  Indicated  during  operation  and  to 
Isolate  the  fault  at  the  Organizational  level  of  maintenance. 

3.2.4. X.4.2. 1  NO-GO  Condition  Detection  -  The  system  off-line  BIT  features  shall  detect  at 

lease _ percent  of  all  NO-GO  condition  occurrences.  (As  applied  at  the  weapon 

system  level  or  as  applied  Independently  at  the  subsystem  level.) 

3.2.4. X.4.2.2  False  Alarms  -  The  number  of  false  alarms  shall  not  exceed _ 

percent  of  all  Indicated  NO-GO  conditions  occurrences  or  alternately  no  more  than 

_ Indicated  NO-GO  condition  occurrences  In  any  24-hour  period  of  system 

operating  time. 

3.2.4. X.4.2.3  Off-Line  BIT  Fault  Detection  -  The  off-line  BIT  fault  detection  capability  shall 
be  designed  to  monitor,  detect,  and  evaluated  faulta  on  all  aystem  or  subsystem 
functions  available  at  the  aystem  or  aubsystem  Interface.  When  a  fault  or  system 
degradation  Is  detected,  the  off-line  BIT  provision  may  determine  the  amount  of 
degradation  and  automatically  branch  Into  the  appropriate  diagnostic  fault  Isolation 
routine. 

3.24.X4.Z4  Off-Line  BIT  Fault  Isolation  -  The  off-line  BIT  fault  Isolation  routines  shall  be 
provided  at  each  fault  detection  point  and  shall  be  automatically  entered  when  a  NO-GO 
la  detected.  The  off-line  BIT  shall  provide  fault  Isolation  to  one  Line  Replaceable  Unit 

_ percent  of  the  time,  fault  Isolation  to _ or  fewer  Line  Replaceable  Units 

_ percent  of  the  time.  In  no  case  shall  the  ambiguity  group  be  greater  than _ 

LRU. 
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3.2.4. X.2.5  Off-Line  BIT  Fault  Isolation  Time  -  The  off-line  BIT  Fault  Isolation  time  shall  be 
consistent  with  the  requirements  of  the  Organizational  level  Mean  Time  To  Repair  (MTTR) 
requirements. 

3.2.4. X.4.3  BIT  Self  Test  -  BIT  self  test  provisions  shall  be  Incorporated  Into  the _ 

system.  The  time  for  the  BIT  self  test  shall  be  less  than _ (and/or  the  duty  cycle  of 

the  BIT  self  test  shall  be _ ).  [The  BIT  failure  rate  shall  be  less  than _ percent 

of  the  prime  system  BIT  failure  Indication  rate.] 

3.2.4. X.4.4  Fall  Safe  Provisions  -  The  circuits  and  devices  which  provide  BIT  and  fault 
Isolation  functions  shall  be  designated  In  such  a  manner  that  failure  of  these  circuits 
and/or  devices  will  not  cause  a  critical  failure  or  unsafe  action  of  the  system. 

3.2.4. X.4.5  Skill  Levels  -  A  personnel  skill  level  of _ Is  required  to  permit  the 

accomplishment  of  all  actions  associated  with  the  fault  Isolation  and 
removal/replacement  of  LRUs  at  the  operatlonal/Organlzatlonal  level.  BIT  provisions 
and,  where  required,  Organizational  level  test  equipment  and  maintenance  procedures 
will  be  used  to  provide  fault  Isolation  within  the  MTTR  specification. 

3.2.4. X.5  Test  Equipment  Interface  -  Signals  shall  be  Includeid  at  the  module  Interface 
which  maximizes  the  similarity  of  built-in  testing  by  the  equipment  and  external  testing 
by  manual  test  equipment  and/or  on  A  TE  systems.  The  system  shall  be  designed  for 

compatibility  for  test  with  the  selected  or  targeted  ATE  (or _ (Insert  test  equipment 

name/designator)).  Maximum  use  shall  be  made  of  operational  pins  to  provide  test 
control  and  access  to  satisfy  the  fault  detectlon/fault  Isolation  requirements  of  external 
test 

3.2.4. X.6  Test  Tolerances  -  Appropriate  tolerances  and  signal  limits  shall  be  established 
In  diagnostic  routines  at  each  level  which  the  system/equipments  are  subject  to  testing 
such  that  false  alarms  and  Retest  Okay  rates  are  minimized. 

3.2.4. X. 7  Technical  Information  Access  Time  -  Average  time  required  tor  the 
maintenance  technician  to  access  maintenance  technical  Information  shall  be  less  then 
_ minutes  at  the  Organizational  level  of  maintenance. 


Configuration  Item  Development  Specification 

A  model  testability  specification  suitable  for  inclusion  in  the  Cl  development 
specification  is  provided  as  follows: 
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3.2.4. X  Testablllty/Dlagnostlc  -  The _ subsystem/ltem  (Insert  name )  shall  be 

designed  for  testability  and  constructed  to  permit  the  status  of  the  subsystem/ltem  and 
the  unambiguous  location  of  faults  to  be  confidently  determined  and  reported  In  a  timely 
fashion. 

3.2.4. X.1  Partitioning  (Functional  Modularity)  •  Subsystems/Items  will  be  partitioned  Into 
Shop  Replaceable  Units  (SRU)  based  on  the  function,  minimum  or  optimum  number  of 
Interconnections,  the  ability  to  fault  Isolate  the  correct  unit  and  the  ability  of  personnel 
with  the  aid  of  support  equipment,  training,  and  technical  manuals  to  fault  Isolate  to  the 
correct  unit. 

3.2.4. X.2  Test  Points  and  Contacts  •  Test  points  snd  contacts  shall  be  conveniently 
located  and  have  safe  access  to  signal  nodes  on  the  unit  under  test,  and  shall  be 
provided  tor  the  measure  or  Injection  of  significant  parameters  for  the  purpose  of 
evaluating  or  troubleshooting  the  circuit  mechanisms.  The  number  and  choice  of 
accessible  nodes  shall  be  sufficient  to  obtain  the  equipment  fault  detectlon/lsolatlon 
requirements  listed  herein. 

3.2.4. X.3  Diagnostic  Capability  -  For  each  level  of  maintenance:  all  diagnostic  resources 
shall  be  Integrated  to  provide  a  consistent  and  complete  diagnostic  capability.  A 
complete  diagnostic  capability  must  Identify  the  diagnostic  resources  that  will  be  used  to 
have  full  FD/FI  coverage.  The  degree  of  diagnostic  automation  shall  be  consistent  with 
the  proposed  personnel  skill  levels  and  maintenance  repair  Items. 

3.2.4. X.4  Built-In  Test  -  Built-In  Test  (BIT)  provisions  shall  be  added  to  the _ 

subsystem/ltem  to  satisfy  system  level  performance  monitoring  and  off-line  BIT 
requirements. 

3.2.4. X.4. 1  On-Line  BIT  Performance  Monitoring  -  The  on-line  BIT  performance 
monitoring  features  shall  be  operative  and  shell  provide  valid  performance  Indications 
prior  to  and  during  operation.  The  performance  monitoring  operation  shall  be  automatic 
and  continuous  shall  be  automatic  and  continuous  and  will  monitor  self-contained  signal 
generating  circuitry.  All  BIT  Implementations  of  this  requirement  shall  be  contained 
within  the  system  or  subsystem  hardware  and  will  not  degrade  mission  performance  at 
any  time. 

3.2.4. X.1.2  NO-GO  Condition  Detection  •  The  system  on-line  BIT  performance  monitoring 

features  shall  detect  at  least _ percent  of  all  NO-GO  condition  occurrences.  (As 

applied  Independently  at  the  subsystem  level.) 

3.2.4. X.4. 1.3  False  Alarms  •  the  number  of  false  alarms  shall  not  exceed _ percent 

of  all  Indicated  NO-GO  condition  occurrences  or  alternately  no  more  than _ 

Indicated  NO-GO  condition  occurrences  In  any  24-hour  period  of  system  operating  time. 
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3.2.4. X.4. 1.4  Performance  Monitor  and  Self-Teat  Data  -  Performance  monitor  and  self-teat 
data  shall  be  transmitted  In  a  manner  such  that  the  transmitted  data  flow  shall  be 
transmitted  In  a  manner  such  that  the  transmitted  data  shall  follow  the  actual  condition 
of  the  system,  that  Is,  a  malfunction  which  corrects  Itself  shall  change  the  fault  data  line 
accordingly. 

3.2.4. X.4.2  Off-Line  BIT  -  The _ subsystem  BIT  provision  shall  furnish  the  means 

for  an  operator  to  Initiate  BIT  tests  at  the  system  level  for  purpose  of  determining  and 
displaying  the  functional  status  of  the  system/subsystems  Including  a  fault  detection  and 
Isolation  capability.  The  Intended  use  of  the  off-line  BIT  test  la  two-fold:  as  a  System 
readiness  test  to  permit  operating  crew  to  accumulate  status  and  fault  Information  an 
opportunity  basis  prior  to  and  during  operations;  and  to  verify  a  fault  Indicated  during 
operation  and  to  Isolate  the  fault  at  the  0-Level  of  maintenance. 

32.4JC4.Z1  NO-QO  Condition  Detection  -  The  system  off-line  BIT  Matures  shall  detect  at 

least _ percent  of  all  NO-QO  condition  occurrences.  (As  applied  Independently  at 

the  subsystem  level.) 

3.Z4.X.4.2.2  False  Alarms  -  The  number  of  false  alarms  shall  not  exceed _ percent 

of  all  Indicated  NO-GO  condition  occurrences  or  alternately  no  more  than _ 

Indicated  NO-QO  condition  occurrences  In  any  24-hour  period  of  system  operating  time. 

3.2.4. X.4.Z3  Off-Line  BIT  Fault  Detection  -  The  off-line  BIT  fault  detection  capability  shall 
be  designed  to  monitor,  detect,  and  evaluate  faults  on  all  system  or  subsystem  functions 
available  at  the  system  or  subsystem  Interface.  When  a  fault  or  system  function 
degradation  Is  detected,  the  off-line  BIT  provisions  shall  determine  the  amount  of 
degradation  and  automatically  branch  Into  the  appropriate  diagnostic  fault  Isolation 
routine. 

3.Z4.X.4.Z4  Off-Line  BIT  Fault  Isolation  -  The  off-line  BIT  fault  Isolation  routines  shall  be 
provided  at  each  fault  detection  decision  point  and  shall  be  automatically  entered  when  s 
NO-QO  Is  detected.  The  off-line  BIT  shall  provide  fault  Isolation  to  one  Shop  Replaceeble 

Unit _ %  of  the  time,  fault  Isolation  to _ or  fewer  Shop  replaceable  Units _ % 

of  the  time. 

3.Z4.X4.2.5  Off-Line  BIT  Fault  Isolation  Time  •  The  off-line  BIT  fault  Isolation  time  shell 
be  consistent  with  the  requirements  of  Mean  Time  To  Repair  ( MTTR )  requirements. 

3.Z4.X.4.3  BIT  Self  Test  •  BIT  self  test  provisions  shell  be  Incorporated  Into  the _ 

subsystem/item.  The  time  for  the  FIT  self  test  shall  be  less  then _ (and/or  the  duty 

cycle  of  the  BIT  self  teat  shall  be _ ).  The  BIT  failure  rate  shall  be  less  than _ % 

of  the  prime  system  BIT  failure  Indication  rate. 

3.2.4. X.4.4  Fall  Safe  Provisions  •  The  circuits  end  devices  which  provide  BIT  and  fault 
Isolation  functions  shall  be  designed  In  such  a  manner  that  failure  of  these  circuits 
and/or  devices  will  not  cause  a  critical  failure  or  unsafe  action  of  the  subsyatem/ltem. 
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3.2.4. X. 6  Skill  Levels  -  A  personnel  skill  level  of _ la  raqulrad  to  parmlt  tha 

accompllahmant  of  all  actions  aaaoclatad  with  tha  fault  Isolation  and 
ramoval/raplacamant  of  SRUa  at  tha  Intarmadlata  maintenance  level.  BIT  provisions,  teat 
equipment  and  maintenance  procedures  will  be  used  to  provide  fault  Isolation  within  the 
MTTR  specification. 

3.2.4JC.7  Test  Equipment  Interface  -  Signals  shall  be  Included  at  the  module  Interfaces 
which  maximize  the  similarity  of  built-in  testing  by  the  equipment  and  off-board  testing 
by  manual  test  equipment  and/or  on  ATE  syatems.  The  system  shall  be  designed  for 
compatibility  for  test  with  target  off-line  automatic  test  equipment  Maximum  use  shall 
be  made  of  operational  pins  to  provide  test  control  and  acceaa  to  satisfy  the  fault 
detection/fault  Isolation  requirements  of  off-board  test 

3.2.4. X.8  LRU  Fault  Detection/Isolation  Requirements  •  The  following  requirements  apply 
to  fault  detectlon/lsolatlon  capability  at  the  Intermediate  level  of  maintenance  using 
automatic  test  resources  (ATE/TPS  and  BIT). 

•  Fault  Isolation  shall  be _ percent  of  all  organizational 

detected  failures. 

-  Average  (or  maximum)  teat  time  for  QO/NO-QO  end-to-end  teats 

shall  be  leas  then _ (minutes/hours). 

-  Maximum  rate  of  false  NO-QO  Indications  resulting  In  Cannot 

Duplicates  and  Retest  Okays  shall  be _ percent  of  all 

Organizational  level  detected  failures. 

-  Fault  Isolation  capability  shall  provide  fault  Isolation  to  one  SRU 

_ percent  of  the  time,  fault  Isolation  to _ or  fewer  SRUs 

_ percent  of  the  time.  In  no  case  shall  the  ambiguity  group 

size  be  greater  than _ SRU. 

-  Average  (or  maximum )  diagnostic  fault  Isolation  time  ahall  be  leas 

than _ (minutes/hours). 

3JL4JX.9  SRU  Fault  Detectlon/lsolatlon  Requirements  -  The  following  requirements  apply 
to  fault  detectlon/lsolatlon  capability  at  the  Depot  level  of  maintenance  using  automatic 
test  resources  (ATE/TPS  and  BIT). 

-  Fault  Isolation  shall  be _ percent  of  all  detected  failures. 

-  Average  (or  maximum)  teat  time  for  QO/NO-QO  end-to-end  teats 

shall  be  less  than _ (mlnutea/houra). 

-  Maximum  rate  of  false  NO-QO  Indications  resulting  In  Reteat 

Okays  shall  be _ percent  of  ad  detected  failures. 

-  Fault  Isolation  capability  shall  provide  fault  Isolation  to  one 

component _ percent  of  the  time,  fault  Isolation  to _ or 

fewer  components _ percent  of  the  time.  In  no  case  shall  the 

ambiguity  group  size  be  greater  than _ components. 

-  Average  (maximum)  diagnostic  fault  Isolation  teat  time  ahall  be 

leas  than _ (mlnutea/houra). 
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CHECKLIST 

Ef  Has  the  Contractor  Program  Manager  worked 
closely  with  the  Government  Program  Manager, 
prior  to  the  issuance  of  the  RFP,  to  assure  the 
diagnostic  requirements  are  feasible, 
cost-effective  and  promote  innovation? 

Does  the  proposal  address  all  the  diagnostic  require 
ments?  Does  it  promote  integration? 

Does  it  promote  innovation? 

E^  Does  the  specification  address  necessary 

FD/FI  requirements  for  each  level  of  maintenance? 

E^  Has  the  concept  of  "diagnostic  growth"  been 
invoked  in  the  diagnostic  specification? 
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WEAPON 
SYSTEM 
ACQ.  PHASE 

OPER. 

REQMTS. 

CONCEPT 

EXPLOR. 

DEM/ 

VAL 

FSD 

PROD/ 

DEPL. 

WEAPON 

SYSTEM 

ACTIVITIES 

A A  A  A 

CONTRACT  AWARD 

DIAGNOSTIC 

actmties 

A  A  A~~A 

c  c  c  c 

PREPARE/UPDATE  DIAGNOSTIC  CAPABILITY  PROGRAM  PLANS 

DIAGNOSTIC  ACTIVITY 


C-  CONTRACTOR  PREPARED 


Program  planning  is  required  to  ensure  that  the  development  and  support  of  the 
diagnostic  capability  is  properly  managed  throughout  the  acquisition  of  a  weapon  system. 
This  planning,  which  is  prepared  by  the  Contractor  Program  Manager,  must  address  how 
this  development  will  be  conducted  to  achieve  this  goal. 

PROCEDURE 

Program  planning  for  the  development  of  the  diagnostic  capability  Is  required 
throughout  the  acquisition  of  the  weapon  system.  It  begins  soon  after  the  award  of  the 
first  developmental  contract  and  is  expanded  and  updated  as  the  development  proceeds. 

The  program  planning  can  take  the  form  of  a  single  Diagnostic  Capability 
Program  Plan  or  can  be  incorporated  in  a  series  of  program  plans  which  are  described  in  a 
number  of  programmatic-type  military  standards.  The  requirements  of  these  planning 
documents  will  be  defined  in  the  contract’s  Statement  of  Work.  To  avoid  unnecessary 
duplication  of  programs  plans,  the  inclusion  of  this  planning  information  in  existing 
documents  is  preferred.  Therefore,  the  guidance  which  follows  addresses  diagnostic 
inputs  to  these  existing  plans,  but  can  be  easily  be  tailored  to  respond  to  the  requirements 
in  the  contract 

The  SEMP  is  the  logical  key  plan,  because  it  reinforces  the  principle  that 
integrated  diagnostics  is  a  systems  engineering  function.  Thus,  when  a  SEMP  is  required, 
the  Government  Program  Manager  will  likely  rely  on  this  plan  as  the  critical  diagnostic 
planning  document.  If  required,  the  Testability  Program  Plan  usually  would  be  included  as 
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an  integral  part  of  the  SEMP  for  testability  is  a  design  consideration  and  is  the  foundation 
of  the  diagnostic  capability. 

GUIDANCE 

Each  of  the  management-type  plans  is  required  during  specific  phases  of  the 
weapon  system  acquisition.  The  following  (Table  2)  is  a  listing  of  these  plans  and  phases 
where  these  plans  are  generally  required.  Normally,  the  initial  version  of  the  SEMP  is 
prepared  during  Concept  Exploration.  An  alternative  is  for  the  government  to  require  the 
preparation  of  the  SEMP  as  part  of  the  contractor’s  response  to  the  Request  for  Proposal 
(RFP). 


TABLE  2  -  APPLICATION  MATRIX 


PLAN  TITLE 


LOGISTIC  SUPPORT 
ANALYSIS  PLAN  (LSAP) 


TESTABILITY 
PROGRAM  PLAN 


RELIABILITY 
PROGRAM  PLAN 


MAINTAINABILITY 
PROGRAM  PLAN 


SUPPORT  PLAN  (ISP) 


SAFETY  PLAN 


AND  EVALUATION 
ER  PLAN  (TEMP) 


PROGRAM  PHASE 


dem/val  FSD 


OR 

ML-CTD-13M 


ML— STD— 882 


DOD—D— 8000.3 


SYSTEM  ENGINEERING  MANAGEMENT  PLAN  (SEMP 


The  requirement  for  a  SEMP,  which  is  composed  of  three  (3)  parts,  is  governed 
by  MIL-STD-499.  Specific  guidance  relating  to  preparation  of  this  plan  follows: 
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PART  I  -  Technical  Program  Planning  and  Control 

This  part  of  the  plan  should  describe  the  contractor(s)  organization  and  internal 
interfaces  required  to  integrate  the  design  of  the  diagnostic  capability  as  an  integral  part  of 
the  system  engineering  process.  The  extent  to  which  integrated  diagnostics  has  been 
institutionalized  within  the  contractor’s  operating  polices  and  procedures  must  be 
addressed.  A  single  individual  shall  be  identified  which  has  the  overall  responsibility  and 
authority  for  implementation  of  the  integration  process.  The  placement  of  this  individual 
within  the  contractor's  organization  is  an  important  issue.  Since  no  two  company 
organizations  are  identical  and  operate  the  same,  there  can  be  no  specific  guidance  on 
this  issue  other  than  saying  the  individual: 

o  Should  have  direct  influence  over  both  the  design  and  supportability 
diagnostic  functions. 

o  Should  have  direct  access  to  the  Contractor  Program  Manager. 

The  review  process  is  used  to  ensure  that  the  integration  of  the  task  is 
accomplished  across  all  involved  functional  disciplines  and  that  an  adequate  feedback 
system  exists  to  redirect  efforts  to  meet  diagnostic  goals  and  requirements.  Where 
subcontractors,  or  teaming  arrangements  with  associate  contractors,  contribute  to  the 
integration  of  the  diagnostic  capability,  describe  these  organizational  interfaces  and  the 
planning  and  control  functions  to  be  implemented  to  assure  a  totally  integrated  effort.  A 
schedule  should  be  established  for  each  of  the  data  deliverables  cited  in  the  Statement  of 
Work. 

PART  II  ■  System  Engineering  Process 

This  part  of  the  plan  should  contain  a  description  of  the  process  to  be  used  in 
meeting  the  overall  program  objectives  and  requirements,  the  general  maintenance 
concept  to  be  used  to  support  the  system/equipment,  and  the  contractor’s  methodology  for 
arriving  at  the  desired  diagnostic  approach.  Analyses  and  trade  studies  should  be 
identified  and  the  proposed  methodology  for  conducting  these  studies  described. 
Reference  to  models  approved  by  the  procuring  activity  must  satisfy  the  methodology 
requirement.  If  not,  these  models/methodologies  should  be  described,  along  with  their 
capabilities  and  limitations.  The  relationship  and  interface  with  the  Logistic  Support 
Analyses  required  by  MIL-STD- 1388-1  should  be  established.  In  addition,  the  plan  should 
include: 


1.  An  integrated  approach  to  the  maintenance  diagnostics  design  that  is  an 
integral  part  of  the  weapon  system/subsystem(s)  design. 

2.  Discussion  of  how  diagnostic  requiremt  ,»  are  to  be  met  and  integrated  with 
each  other  in  the  overall  weapon  system  design.  This  shall  Include 


1-39 


DIAGNOSTIC  CAPABILITY  PROGRAM  PLANNING 


REQUIREMENT  #1.3 


procedures  for  identifying  deficiencies,  needed  actions,  and  corrective 
measures. 

3.  Discussion  of  how  diagnostic  elements  are  integrated  with  each  other  into 
the  cost-effective  achievement  of  primary  maintenance  goals  (e.  g.,  100% 
unambiguous  fault  isolation  capability). 

PART  III  -  Engineering  Specialty  Integration 

This  part  should  include  a  detailed  description  of  the  integrated  diagnostics 
interrelationships  involving  human  engineering,  personnel,  safety,  reliability,  training, 
logistics,  product  assurance,  maintainability,  etc.,  and  their  integration  with  the  system 
engineering  process.  The  plan  should  address  the  need  for  combined  demonstration 
programs  (e.  g.,  reliability,  maintainability). 

LOGISTIC  SUPPORT  ANALYSIS  PLAN 

The  Logistic  Support  Analysis  Plan  (see  MIL-STD-1 388-1)  should  define  the 
interface  between  the  analysis  being  conducted  to  define  the  specification  for  the 
diagnostic  capability  and  the  LSA. 

RELIABILITY  PROGRAM  PLAN 

Specifically,  the  Reliability  Program  Plan  should  address  the  conduct  of  the 
Failure  Modes,  Effects,  and  Critically  Analysis  (FMECA),  as  the  basis  for  initial  diagnostic 
design.  In  addition,  the  reliability  modeling  task.  Task  201 ,  MIL-STD-785,  should  take  into 
count  fault-tolerant  design  and  its  relationship  meeting  diagnostic  goals  utilizing 
redundancy. 

MAINTAINABILITY  PROGRAM  PLAN 

The  Maintainability  Program  Plan  is  the  basic  planning  document  for  assuring 
that  diagnostic  requirements  are  met.  Each  of  the  MIL-STD-470  200-series  tasks  has  a 
direct  interface  with  the  design  of  the  diagnostic  capability.  In  addition,  Task  301, 
Maintainability  Demonstration,  is  the  basic  demonstration  task  for  both  testability  and 
diagnostics. 

INTEGRATED  SUPPORT  PLAN 

This  is  the  formal  planning  document  for  logistic  support.  It  must  reflect  how  all 
of  the  diagnostic  elements  will  be  provided  and  supported. 
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SYSTEM  SAFETY  PLAN 

The  System  Safety  Plan  (MIL-STD-882)  should  provide  diagnostic  inputs  that 
impact  the  determination  and  identification  of  diagnostic  requirements  for  detecting 
potential  safety  problems.  This  performance  monitoring  analysis  should  be  closely  tied  to 
the  FMECA. 

HUMAN  ENGINEERING  PROGRAM  PUN 

The  Human  Engineering  Program  Plan  needs  to  address  the  technician’s 
role/interface  with  the  entire  weapon  system  diagnostic  capability,  including  the  time 
required  to  access  technical  information  from  whatever  media  is  used.  Careful  attention 
must  be  paid  in  planning  to  have  technicians  evaluate  the  entire  diagnostic  capability  (at 
all  maintenance  levels)  during  test  and  evaluation. 
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CHECKLIST 

O'  Does  the  SEMP  address  design  of  the  diagnostic  cap¬ 
abilities  as  an  integral  part  of  the  system  engineering 
process  (i.e.,  emphasis  on  Parts  I  and  II  of  tne 
SEMP,  as  opposed  to  Part  III)? 

Does  Part  III  of  the  SEMP  require  combined  "ility" 
and  logistic  demonstrations? 

O'  Are  the  diagnostic  implications  included  ,in  the 
various  "ility"  and  logistic  plans? 

O'  Do  the  various  plans  place  major  emphasis  on 
design  for  testability? 

Is  there  a  single  individual  assigned  full  diagnostic 
responsibility?  With  "sign  off"  authority? 
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WEAPON 
SYSTEM 
ACQ.  PHASE 

OPER. 

REQMTS. 

CONCEPT 

EXPLOR. 

DEM/ 

VAL 

FSD 

PROD/ 

DEPL. 

WEAPON 

SYSTEM 

AcrivmES 

Z 

L  L 

APPRC 

PRC 

x  i nr 

- - - - ^LOGISTIC 

>VAL  TO  REVIEW 

CEED 

DIAGNOSTIC 

ACTIVITIES 

A  A  AAA 

SCP  DCP  DCP 

UPDATE 

DIAGNOSTIC  ACTIVITY 


Prior  to  Milestones  I  through  V  (DoD  Instruction  5000.2),  the  preparation  of  a 
paper  by  the  Government  Program  Manager  is  required  to  summarize  the  results  of  the 
acquisition  and  deployment  of  a  major  weapon  system.  Prior  to  Dem/Val,  a  System 
Concept  Paper  (SCP)  is  required.  Prior  to  FSD,  preparation  of  a  Decision  Coordinating 
Paper  (DCP)  is  required.  An  update  of  the  DCP  is  required  prior  to  Milestones  III,  IV,  and 
V.  The  SCP  and  the  DCPs  for  Milestones  II  and  III  are  required  to  secure  approval  to 
proceed  to  the  next  acquisition  phase.  Milestone  IV  is  a  logistic  readiness  and  support 
review,  which  is  conducted  one  or  two  years  after  deployment  to  assure  that  operational 
readiness  and  support  objectives  are  achieved.  Milestone  V  is  a  major  upgrade  or  system 
replacement  decision,  which  also  requires  an  updating  of  the  DCP.  Diagnostic  Issues 
should  be  addressed  in  these  documents. 

PROCEDURE 


DoD  Instruction  5000.2  delineates  the  need  and  the  format  for  both  an  SCP  and 
a  DCP.  It  is  likely  that  this  documentation  will  address  diagnostic  issues.  Although  this 
type  of  documentation  is  required  only  for  major  weapon  systems,  similar  documentation 
may  be  required  by  the  individual  services  at  significant  milestones.  Thus  the  following 
guidance  can  apply  to  all  weapon  systems  and  equipment.  It  is  also  important  to  note  that 
Milestone  IV,  being  a  logistic  readiness  and  support  review,  coincides  with  the  projected 
maturation  schedule  for  the  diagnostic  capability.  Thus  the  Maturation  Program  Plan  will 
probably  relate  to  this  milestone. 
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These  documents  are  almost  always  prepared  by  the  Government  Program 
Manager.  The  following  guidance  is  included  in  this  document  to  promote  understanding 
by  the  Contractor  Program  Manager  of  what  is  required  to  successfully  justify  a  weapon 
system  development. 

GUIDANCE 


The  format  for  the  SCP  and  the  DCP  is  identical.  However,  the  SCP  is  primarily 
concerned  with:  (1)  program  alternative  tradeoffs;  (2)  performance/cost  and  schedule 
tradeoffs,  including  the  need  for  a  new  development  program  vs.  buying  or  adapting 
existing  U.  S.  or  Allied  military  or  commercial  systems;  (3)  appropriateness  of  the 
acquisition  strategy;  (4)  prototyping  of  the  system  or  selected  system  components;  (5) 
affordability  and  life  cycle  costs;  (6)  potential  common-use  solutions;  and  (7)  cooperative 
development  opportunities.  Thus  diagnostic  inputs  to  the  SCP  will  most  likely  address 
these  factors. 

On  the  other  hand,  the  DCP  requirements  for  Milestone  II,  Full-Scale 
Development  Decision,  will  normally  address:  (1)  affordability,  in  terms  of  program  cost  vs. 
the  military  value  of  the  new  or  improved  system  and  its  operational  suitability  and 
effectiveness;  (2)  program  risk  vs.  benefit  of  added  military  capability;  (3)  planning  for  the 
transition  from  development  to  production,  which  will  include  independent  producibility 
assessments  (hardware/software/data  bases);  (4)  realistic  industry  surge  and  mobilization 
capacity;  (5)  factors  that  impact  program  stability;  (6)  potential  common-use  solutions;  (7) 
results  from  prototyping  and  demonstration/validation;  (8)  milestone  authorization;  (9) 
manpower,  personnel,  training,  and  safety  assessments;  (10)  procurement  strategy 
appropriate  to  program  cost  and  risk  assessments;  (11)  plans  for  integrated  logistics 
support  (DoD  Directive  5000.39);  (12)  affordability  and  life  cycle  costs;  and  (13)  associated 
command,  control,  communications,  and  intelligence  requirements,  including 
communications  security. 

The  DCP  requirements  for  Milestone  III,  Full  Rate  Production  Decision  will 
normally  address:  (1)  results  of  completed  operational  test  and  evaluation;  (2)  threat 
validation;  (3)  production  or  construction  cost  verification;  (4)  affordability  and  life  cycle 
costs;  (5)  the  production  and  deployment  schedule;  (6)  reliability,  maintainability,  and 
plans  for  integrated  logistics  support  (DoD  Directive  5000.39);  (7)  producibility,  as  verified 
by  an  independent  assessment  (DoD  Directive  5000.38);  (8)  realistic  industry  surge  and 
mobilization  capacity;  (9)  multiyear  procurement  or  milestone  authorization;  (10) 
manpower,  personnel,  training,  and  safety  requirements;  (11)  cost  effectiveness  or  plans 
for  competition  or  dual  sourcing;  and  (12)  associated  command,  control,  communications, 
and  intelligence  requirements,  including  communications  security. 

Primary  considerations  for  Milestone  IV,  Logistic  Readiness  and  Support 
Rev!ew,  are:  (1)  logistic  readiness  and  sustainability  (peacetime  and  wartime);  (2)  weapon 
support  objectives;  (3)  the  implementation  of  integrated  logistics  support  plans,  per  DoD 
Directive  5000.39;  (4)  the  capability  of  logistics  activities  (i.  e„  supply,  transportation,  etc), 
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facilities,  training,  and  manpower  to  provide  support  efficiently  and  cost  effectively;  (5) 
disposition  of  displaced  equipment;  and  (6)  affordability  and  life  cycle  costs.  This  should 
coincide  with  the  completion  of  the  diagnostic  capability’s  maturation  period. 

Milestone  V,  Major  Upgrade  or  System  Replacement  Decision,  is  concerned 
with:  (1)  capability  of  the  system  or  facility  to  continue  to  meet  its  original  or  evolved 
mission  requirements;  (2)  the  potential  necessity  of  modifications  and  upgrades  to  ensure 
that  mission  requirements  are  met  and  that  the  useful  life  is  extended;  (3)  changes  in 
threat  that  require  increased  capability  or  utility;  (4)  changes  in  technology  that  present  the 
opportunity  for  a  significant  breakthrough  in  system  worth;  and  (5)  disposition  of  displaced 
equipment.  A  significant  question  to  be  decided  at  this  point  is  whether  deficiencies  are 
critical  enough  to  warrant  major  modification,  retirement,  and/or  new  start  considerations. 
Feedback  on  the  performance  of  the  diagnostic  capability  is  an  important  factor  in  this 
decision. 


The  above  guidance  relates  to  all  of  the  information  required  in  preparing  SCPs 
and  DCPs.  Each  of  these  items  will  normally  be  addressed  by  the  Government  Program 
Manager  in  relation  to  the  diagnostic  capability,  as  appropriate.  Much  of  the  data  required 
to  provide  this  information  will  be  an  output  from  the  various  studies  conducted  by  the 
contractor  as  discussed  in  this  guide  under  Requirement  #2. 
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CHECKLIST 

Am  I  furnishing  the  Government  Program  Manager 
with  the  proper  information  for  him  to  adequately 
justify  his  program? 


REQUIREMENTS 


REQUIREMENT  #2 


ESTABLISHING  AND  ALLOCATING  DIAGNOSTIC  REQUIREMENTS 


OVERVIEW 

Good  diagnostics  and  testability  are  based  on  the  ability 
to  properly  establish  diagnostic  requirements,  which  are 
in  turn  based  on  weapon  system  mission,  the  system’s 
sustainability,  operational,  and  support  requirements 
and  the  allocation  of  these  requirements  at  system, 
subsystem,  and  unit  levels.  Lack  of  appropriate  attention 
to  this  process  results  in  diagnostic  designs  with 
questionable  basis  and  justification.  Unfortunately,  this 
process  has  not  been  transformed  from  an  art  to  a 
rigorous  methodology.  An  integrated  series  of  proven 
tools  does  not  exist  and  thus  the  quality  of  the  process 
depends  on  the  expertise  of  the  persons  performing  this 
function.  The  system  is  further  complicated  by  the 
advanced  weapon  system  architecture  which  is  now 
being  applied.  This  architecture  involves  complex 
redundancy,  reconfigurable  elements,  and 
configurations  which  allow  graceful  degradation.  A 
proper  allocation  methodology  is  an  integral  part  of 
logistic  support,  reliability,  and  maintainability  analyses 
and  is  based  on  the  weapon  system’s  mission  scenario 
and  performance  requirements.  The  analyses  are  an 
iterative  process,  which  extend  over  the  acquisition 
phases  and  often  into  deployment  of  the  weapon 
system,  implementation  of  these  analyses  are  normally 
the  responsibility  of  the  Contractor  Program  Manager, 
with  the  results  reviewed  by  the  Government  Program 
Manager. 


TESTABILITY/ 

DIAGNOSTICS 


PROGRAMMATIC 


DESIGN 


ASSESSMENT 


REVIEWS 


EVALUATION 


MATURATION 


IMPORTANT  CONSIDERATIONS  TO  BE  ADDRESSED 

Reqmt. 

2.1  Translate  mission  and  performance  requirements  into  diagnostic 
requirements. 

2.2  Allocate  diagnostic  requirements  to  system,  subsystem  and  unit  elements. 

2.3  Optimize  the  mix  of  diagnostic  elements. 

2.4  Assess  the  risk  of  each  diagnostic  alternative. 
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WEAPON 
SYSTEM 
ACQ.  PHASE 

OPER. 

REQMTS. 

CONCEPT 

EXPLOR. 

DEM/ 

VAL 

FSD 

PROD/ 

DEPL. 

WEAPON 

SYSTEM 

ACTIVmES 

Z S  A 

" - - - ^ 

CONTRACT 

AWARD 

DIAGNOSTIC 

ACTIVITIES 

A  A 

DIAGNOSTIC  UPDATE 

REQUIREMENTS 

DIAGNOSTIC  ACTIVITY 

Diagnostic  requirements  are  identified  in  the  Concept  Exploration  Phase  from  an 
analysis  of  the  prime  system  mission  and  operational  requirements. 

PROCEDURE 


The  generation  of  weapon  system  operational  requirements  is  usually 
performed  by  the  government  from  mission  studies  and  analyses  based  on  the  Statement 
of  Need  for  a  weapon  system.  The  translation  of  those  requirements  and  weapon  system 
performance  characteristics  Into  diagnostic  requirements  which  are  included  in  the  system 
specification  produced  as  a  result  of  Concept  Exploration.  The  tasks  involved  in  translating 
these  requirements  may  be  performed  by  the  contractors  or  the  government  depending 
upon  the  acquisition  process  selected  during  Concept  Exploration.  For  "in-house" 
programs,  this  task  is  performed  by  government  engineers.  Frequently,  however,  the 
translation  of  mission  and  performance  characteristics  into  diagnostic  requirements,  the 
selection  and  integration  of  the  diagnostic  elements  to  meet  these  requirements,  the 
allocation  of  these  requirements  to  subsystem  and  unit  level,  and  the  assessment  of  risk  is 
performed  by  the  weapon  system  contractors. 

The  proper  implementation  of  this  task  is  that  it  be  performed  in  conjunction  with 
the  system  engineering  and  logistic  support  analysis  process  and  include  synthesis  and 
analysis  of  the  various  mixes  of  resources  which  make  up  a  total  diagnostic  subsystem. 
The  diagnostic  requirements  analysis  process  involves  the  development  of  a  strategy  for  a 
comprehensive  diagnostic  capability  including  a  mix  of  resources  to  be  defined  for 
roviding  FD/FI  capability  at  each  level  which  the  system  is  subject  to  maintenance. 
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In  order  to  translate  mission  and  operational  requirements  to  diagnostic 
capability,  it  is  important  to  postulate  a  "diagnostic  subsystem."  Characteristics  defining 
the  capability  of  the  "diagnostic  subsystem”  represent  the  results  of  the  translation.  In 
other  words,  one  must  change  mission  requirements  into  diagnostic  capability 
requirements  in  order  to  successfully  complete  this  task. 

The  diagnostic  elements  constituting  the  diagnostic  subsystem  include 
embedded  diagnostics,  support  equipment  at  all  levels  of  maintenance,  technical  data  in 
all  its  forms,  and  personnel  numbers  and  required  skill  levels. 

In  order  to  be  responsive  to  weapon  system  mission  and  performance 
requirements,  it  is  essential  that  the  translations  start  by  reviewing  all  the  requirements 
documentation  and  studies.  The  key  document  is  the  Statement  of  Need  which  contains 
the  weapon  system  mission  and  operational  requirements.  Also  important  is  the  prime 
system  architecture  concept  which  is  an  essential  element  in  the  translation  since  many 
architectural  concepts  contain  an  inherent  diagnostic  capability  that  must  be  identified  and 
addressed  early  in  the  analysis  process. 

There  are  two  key  factors  which  will  influence  the  translation  of  weapon  system 
mission  and  operational  requirements  into  diagnostic  requirements.  They  are: 

o  Specific  requirements  as  spelled  out  in  the  Statement  of  Need 

o  Available  technology. 

Analysis  of  these  specific  requirements  will  translate  requirement  for  the 
diagnostic  capability  as  well  as  constraints  on  the  diagnostic  subsystem  dictated  by  the 
operational  parameters.  The  technology  will  impact  the  Inherent  diagnostic  capability  of 
the  prime  system  architecture  as  well  as  impact  the  assessment  of  risk  of  the  final 
diagnostic  subsystem  implementation. 

Based  upon  the  above  analyses,  translation  of  mission  and  operational 
requirements  to  a  diagnostic  capability  will  result  in  a  preliminary  set  of  diagnostic 
quirements  for  the  entire  diagnostic  subsystem.  The  optimum  mix  of  diagnostic 
elements  which  constitute  the  diagnostic  capability  will  follow  the  requirements  allocation 
to  the  weapon  system,  subsystem  and  unit  levels. 

During  the  Demonstration/Validation  Phase  and  Full-Scale  Development  Phase 
the  detailed  trade  studies  will  formally  optimize  the  diagnostic  element  mix  and  provide 
implementation  specifications  for  the  diagnostic  subsystem  to  be  produced.  This  process 
is  obviously  iterative  but  most  dependent  upon  a  thorough  job  of  mission  and  performance 
requirements  analysis  and  initial  translation  into  diagnostic  requirements. 

For  example,  the  Dem/Val  Phase  may  result  in  a  System  Specification  only,  with 
the  allocation  of  the  system  requirements  to  be  performed  and  redefined  in  the  FSD 
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Phase.  For  some  less-than-major  systems,  Dem/Val  Phase  may  be  bypassed  altogether. 
In  both  of  these  cases,  both  the  system-level  specifications  may  be  developed  during  FSD 
Phase.  The  analyses  described  within  this  section  should  be  performed  at  appropriate 
points  based  upon,  and  commensurate  with,  the  level  of  detail  achieved  in  the  definition  of 
the  system  and  the  definition  of  the  support  and  maintenance  concepts  for  the  system. 

GUIDANCE 


Currently  there  is  no  formal  DoD  model  for  translating  mission  and  operational 
requirements  into  a  diagnostic  capability.  However,  using  system  engineering  approaches 
defined  in  MIL-STD-499,  the  contractor  can,  indeed,  develop  an  initial  set  of  diagnostic 
subsystem  requirements  which  are  traceable  to  weapon  system  requirements,  weapon 
system  priorities,  and  available  technology. 

Success  in  translating  mission  and  operational  requirements  into  diagnostic 
requirements  is  embodied  in  the  ability  to  develop  higher  order  measures  for  defining 
weapon  system  characteristics  that  relate  to  fault  detection  and  fault  isolation  parameters. 

Typical  weapon  system  characteristics  which  must  be  evaluated  include  the  following: 


Probability  of  Mission  Success 

Availability 

Utilization  Rate 

Population 

Turnaround  Time 

Threat 

Mobility 

Safety 

Alert 


Deployment 

Basing 

Weight 

Repair  Concept 

Personnel 

Training 

Cost 

Etc. 


The  foi.owing  weapon  system  priorities  are  of  major  concern: 

War  fighting  capability 

Survivability 

Mobility 

Manpower 

Life  Cycle  Cost. 


During  the  Concept  Exploration  Phase,  mission-oriented  measures  are 
overriding  for  diagnostic  requirements  generation.  Resource  criteria  (manpower,  cost, 
facilities,  etc.)  become  significant  during  synthesis  of  specific  diagnostic  element  mixes. 

The  mission  data  to  be  collected  and  considered  for  generating  the  diagnostic 
requirements  is  as  follows: 
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o  Mission  scenarios  definition  (prioritized  in  order  of  criticality) 

o  Mission  rate/length 

o  Mission  operation  (continuous  vs.  intermittent) 

o  Mission  phases 

o  Time  demands  and  operational  constraints  per  mission  phase 

o  Subsystem/function  utilization  per  mission  phase  (survivability  or  safety 
critical) 

o  Functions/failures  impacting  personnel  safety 

o  Functions/failures  impacting  system/equipment  safety  (sustainability  or 
mission  critical) 

o  Functions/failures  impacting  mission  success  (per  mission  phase). 

A  key  diagnostic  parameter  to  be  determined  through  the  analysis  of  mission 
requirements  is  the  maximum  failure  latency  per  operating  function  for  each  mission 
phase.  This  parameter  will  drive  the  fault  detection  requirements  which,  in  turn,  serve  as 
the  basis  for  BIT  design.  Failure  latency  is  the  elapsed  time  between  fault  occurrence  and 
failure  indication.  Maximum  failure  latency  is  the  maximum  allowable  time  between  the 
occurrence  of  a  fault  and  the  reporting  or  "handling”  of  the  failure.  As  a  simplistic  example, 
if  a  fire  control  system  fault  occurs,  and  the  fire  control  system  function  is  highly  critical  to 
mission  success,  then  the  maximum  failure  latency  will  be  very  small  --  perhaps  expressed 
in  microseconds  or  nanoseconds.  The  fault  detection  (FD)  time  requirement  will  reflect  the 
failure  latency  factor  ~  thereby  driving  the  BIT  technique  to  provide  concurrent 
performance  monitoring.  Fault  tolerance  through  redundancy  may  be  required  or 
considered.  This  simplistic  example  is  made  more  complex  by  factoring  in  the  time 
demands  per  mission  phase  of  the  fire  control  system.  It  is  made  still  more  complex  by 
factoring  in  operating  anomalies  and  intermittents  into  the  FD  requirements. 

In  the  definition  of  diagnostic  requirements,  it  is  important  to  note  that  the 
diagnostic  capability  is  made  up  of  the  inherent  diagnostic  capability  of  the  prime  system, 
as  well  as  added  diagnostic  elements.  It  is  therefore  important  that  diagnostic  analysis  be 
integral  to  the  prime  weapon  system  engineering  process,  since  performance  and  support 
parameters  can  no  longer  be  isolated  from  design. 

The  prime  configuration  represents  a  performance  capability.  The  mission 
requirements  can  be  related  directly  to  the  configuration  by  analysis  of  the  behavior  of  the 
utilized  configuration  items  over  the  time  demands  imposed  by  mission.  A  representation 
of  performance  over  time  can  be  easily  presented  to  management  for  setting 
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requirements.  This  measure  is  referred  to  as  P  (Performance  time  dependency),  which  is 
described  in  MIL-HDBK-338.  P  can  be  calculated  and  plotted  using  equations  for  mission 
reliability  in  MIL-STD-756. 

Operational  constraints  also  must  be  addressed.  The  checklist  below  presents 
the  operational  data  to  be  collected  and  considered  in  diagnostic  requirements  analysis. 

o  Environmental  conditions  (temperature,  rain,  dirt,  salt  spray,  etc.).  Applies  to 
both  the  prime  system  and  support  equipment 

o  Operating  locations  (dispersed  vs.  centralized) 
(remote/accessible/inaccessible) 

o  Space  limitations  (for  personnel  and/or  test  equipment) 

o  Mobile  or  fixed  maintenance  facilities 

o  Independent  operation  or  part  of  a  battle  group 

o  Manpower  constraints  (number  and  skill  levels). 

The  constraints  under  which  a  weapon  system  will  operate  must  be  identified 
and  evaluated  in  terms  of  the  impact  on  testability  requirements.  System  design  and 
supportability  factors  must  take  into  account  these  constraints.  Operating  constraints  will 
often  drive  the  diagnostic  strategy  to  use  of  embedded  versus  external  test  resources. 

Prime  System  Architecture/Configuration 

Data  must  be  collected  on  the  architecture  and  configuration  alternatives  of  the 
prime  system  to  be  developed  with  respect  to  partitioning,  interconnections  and  flow  as 
input  to  the  testability  requirements  analysis.  The  architectures  under  consideration  will 
have  inherent  characteristics  which  may  support  or  impede  diagnostics.  The 
performance  capability  of  alternative  prime  system  architectures  must  be  evaluated 
against  the  mission  requirements,  time  phases  and  equipment  utiiization/demands. 

It  is  useful  for  this  evaluation  to  plot  curves  of  capability  vs.  time  demands 
imposed  by  the  mission.  The  resulting  P  (Performance  over  Time)  curve  can  include 
resource  constraints  (spares,  personnel)  and  operational  constraints  (maximum  allowable 
repair  time). 

The  following  prime  system  configuration  data  should  be  collected  for  input  to 

this  step: 

o  Work  Breakdown  Structure  (MIL-STD-881) 
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o  List  of  government  furnished  equipment/ 

off-the-shelf  equipment/  non-developmental  items 
(for  above,  item  or  candidate  item) 
o  Prime  system  architecture  alternatives 
o  Initial  failure  rate  projections  and  characterization 
o  Fault-tolerant  or  redundant  functions 
o  Technologies  to  be  used  (if  known) 
o  Level  of  integration  vs.  autonomy. 

Based  upon  analysis  of  architectures  under  consideration,  high-level  diagnostic 
opportunities  should  be  identified.  This  includes  incorporation  of  a  test  and  maintenance 
bus,  fault-tolerant  design  coordination,  system-level  diagnostic  resources  -  such  as  data 
acquisition/collection  subsystems  or  embedded  adaptive  diagnostic  subsystem  and  use  of 
standard  diagnostic  connections  and  interfaces. 

Diagnostic  inputs  must  be  made  within  the  system  engineering  process  prior  to 
the  final  selection  of  the  prime  system  architecture. 

Evaluate  Technology  Opportunities 

Advanced  diagnostic  technology  opportunity  or  implications  must  be  identified 
based  on  the  following  areas: 

o  Baseline  comparison  system  major  drivers,  supportability  problem  areas, 
targets  of  improvement 

o  Incorporation  of  LSI,  VLSI,  VHSIC,  expert  system  or  other  advanced  design 
technology  in  system 

o  Need  to  improve  requisite  operational  capability  having  no  prior  design 
solution. 

Examples  of  advanced  diagnostic  technology  opportunities  which  may  be 
exploitable  on  the  new  system  include: 

o  Expert  system  based  maintenance  aids 

o  Test  and  Maintenance  bus  concepts 

o  "Smart"  BIT  techniques 

o  Adaptive  diagnostic  subsystems 

o  Prognostics  concepts 
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o  Automated  technical  information  authoring 
o  Advanced  packaging  techniques 

o  Advanced  instrumentation  (stimulus  and  measurement)  technologies 

o  Automatic  capture  of  CAD  data  for  diagnostics  generation. 

Upon  determination  of  advanced  technology  applications,  inputs  must  be  made 
to  the  design  engineering  effort  regarding  design  constraints  related  to  the  above 
concepts. 

Diagnostic  Element  Constraints 

In  order  to  define  specific  diagnostic  characteristics  and  requirements  of 
the  system  or  to  further  "close  in"  the  envelope  within  which  tradeoffs  are  conducted, 
diagnostic-related  constraints  are  established.  This  includes  constraints  placed  on 
built-in  test  design  attributes  and  functions,  testability  constraints  and  test  equipment 
constraints.  This  may  also  include  broader  diagnostic-related  constraints,  such  as 
page  count  of  technical  information  or  maintenance  technician  skill-levels.  These 
constraints  are  driven  by  mission  requirements,  design,  operation  and  support 
characteristics,  or  standardization  policies  imposed. 


Sample  diagnostic-related  constraints  are  provided  below. 


Driving  System  Requirement 
Mission  Requirement 
Mobility 

Continuous  Operation 

Sustainability 

Reconfigurability 

Standardization  Imposed 

Standard  Test  Equipment 

Standard  Bus 
GFE 


Resulting  Diagnostic  Constraint/Requirement 


Test  Equipment  Size/Weight 

BIT  Interface  Planned  Maintenance  Duty 

Cycle 

Redundancy 
Fault-Tolerant  Design 


Standard  Diagnostic  Connectors 
Controllability,  Observability 
Interface  to  UUT 
Interface  DesigrVProtocol 
Bit  Design/Capabilities 
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Design  Characteristics 

Power  Availability 
System  Weight 
System  Size 


Memory  Limitation 
Operating  System  Char. 
Cost 


BIT  Power  Consumption 

BIT  and  Test  Connector  weight 

Volume  of  BIT  Circuitry  and  Test  Connectors 

(Real  estate  available  for  BIT  circuitry) 

Volume  required  for  increased  modularity 
Memory  allocatable  to  BIT  functions 
Software  BIT  function  constraints 
Cost  of  additional  hardware  required  for  BIT 
and  testability 


Establish  Diagnostic  Objectives 

Analysis  of  weapon  system  data  ascertained  must  be  performed  to  identify 
diagnostic  objectives  based  on  system  requirements.  Diagnostic  objectives  to  be 
considered  include: 


o  BIT  FD/FI  requirements  to  support  preliminary  maintenance  concept 

-  Repair  Times 

-  Reconfigurability 

-  Deferred  Maintenance 

-  Fault  Tolerance 

o  BIT  requirements  to  support  system  confidence  checks 
o  Requirement  to  deal  with  intermittent  faults  or  operational  anomalies 
o  Prime  system  architecture  testability  opportunities 
o  GFE  testability  factors/constraints 
o  Requirements  for  vertical  testability. 


Examples  of  typical  objectives  to  be  established  at  this  point  are  provided 

below. 


SAMPLE 

DRIVING  SYSTEM  FACTOR  DIAGNOSTIC  OBJECTIVE 


Maximum  Acceptable  Failure  Latency — => 

Mission/Safety  Critical  Function - => 

MTTR,  Spares - =» 

Manpower  and  Skill  Levels - => 

GFE  Constraints - ^ 

Fault-Tolerant  Design  Coordination — =* 

2  Level  Maintenance - => 

Life  Cycle  Cost  Priorities - => 

Minimize  RTOK  Rate  Between - =* 

Maintenance  Levels 


Fault  Detection  Time 
Performance  Monitoring 
Fault-Isolation  Level 
BIT  Fault-Isolation  Level 
System-Level  BIT  Requirements 
Performance  Monitoring 
Ambiguity  Group  Size/ATE  Size  & 
Weight 

Reliance  on  Embedded  Diagnostics 
Utilize  Compatible  Test  Equipment, 
Techniques,  Tolerances 
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Initial  diagnostic  requirements  result  from  analysis  of  weapon  system 
characteristics,  prioritized  as  needed,  on  diagnostic  elements.  It  is  convenient  to  partition 
the  diagnostic  elements  as  embedded  and  external. 

Some  of  the  tradeoffs  to  be  made  for  generating  embedded  and  external 
diagnostic  requirements  include: 

o  Functions  and  level  of  built-in  test  vs.  external  diagnostics 

o  Functional  vs.  parametric  testing 

o  Built-in  test  fault  detection 

-  Concurrent  performance  monitoring 

-  Periodic  BIT  routines 

-  Operator  initiated  BIT  routines 

o  Level  of  diagnostic  capability  at  each  level  of  maintenance  (e.g.,  detect  95% 
of  faults;  isolate  to  3  LRUs;  within  30  minutes) 

o  Diagnostic  elements  to  be  used  at  each  level  of  maintenance  (e.g.,  test 
equipment,  technical  information  and  maintenance  aids,  training  and  skill 
levels). 

Once  the  level  of  built-in  test  is  established,  a  maintenance  workload  generated 
by  operational  and  failure  rate  data  can  be  projected.  At  this  point,  detailed  tradeoffs  can 
be  performed  regarding  the  optimization  of  testability,  including  level  of  diagnostic 
capability  at  each  level  of  maintenance,  and  the  effectiveness  and  efficiency  of  the  mix  of 
diagnostic  elements  to  be  used  at  each  level  of  maintenance.  A  baseline  comparison 
system  is  used  to  project  failure  data.  The  requirements  that  need  to  be  established  are 
outlined  below: 


Embedded  Diagnostic  Requirements 

o  System  Integrated  Test  (SIT)  requirement  for  monitoring  of  mission-critical 
functions  and  functions  affecting  personnel  safety  (derived  from  maximum 
allowable  failure  latency) 

o  BIT/SIT  requirements  to  support  operational  constraints 

o  Requirement  to  deal  with/handle  intermittents/anomalies 

o  BIT/SIT  requirements  to  support  system  confidence  checks 

o  Prime  system  architecture,  testability  opportunities,  and  GFE  testability 
factors/constraints 
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o  BIT  requirements  to  support  preliminary  maintenance  concept,  based  on: 

-  Level-of-repair  analysis 

•  Manpower  available 

-  Skill  levels  available/required 

-  Deferred  maintenance  goals 

-  Repair  times  (driving  fault  isolation  time) 

*  Sparing  concepts  (driving  fault  isolation  levels) 

-  Standardization  requirements/goals  (test  equipment,  personnel 
qualifications) 

o  Requirement  to  provide  handshake  to  external  diagnostic  resources  (vertical 
testability,  vertical  diagnostics). 

External  Diagnostic  Requirements  (Support  Equipment,  Technical  Data,  and 
Personnel) 

OPERATIONAL/ORGANIZATIONAL  MAINTENANCE  LEVEL: 

o  Requirement  for  elements  to  optimize  interface/utilization  of  embedded 
diagnostic  elements 

o  Define  FD/FI  functions  to  satisfy  O-Level  maintenance  operations  (driven  by 
inputs  from  operational  constraints  and  preliminary  maintenance  concept), 
based  on: 

-  Minimization  of  unnecessary  removals 
>  Mobility  requirements/space  available 

-  Level-of-repair  analysis 

-  Sustainability 

-  Manpower  available 

-  Skill  levels  available/required 

-  Repair  times 

-  Sparing  concepts 

-  Standardization  requirements/goals 

o  O-Level  technical  information  (including  maintenance  aids) 
o  O-Level  test  equipment 

-  Manual  test  equipment 

-  Automatic  test  equipment  and  test  programs 

-  Portable  maintenance  aids 

o  O-Level  training  requirements  to  support  skills  required 


2-12 


TRANSLATING  MISSION  REQUIREMENTS  REQUIREMENT  #2.1 


-  On-the-job  training 

-  Formal  school  training 

o  O-Level  data  acquisition/collection  system  (and  data  management) 

o  Requirements  to  provide  O-Level  handshake  to  1-Level  diagnostic  elements 
(vertical  testability,  vertical  diagnostics) 

INTERMEDIATE  MAINTENANCE  LEVEL: 

o  Define  FD/FI  functions  to  satisfy  1-Level  maintenance  operations  based  on: 

-  Minimization  of  unnecessary  removals 

-  Mobility  requirements/space  available 

-  Level-of-repair  analysis 

-  Sustainability  of  spares  pipeline 

-  Manpower  available 

-  Skill  levels  available/required 

-  Repair  times 

-  Sparing  concepts 

-  Standardization  requirements/goals 

o  1-Level  technical  information  requirements  (including  maintenance  aids) 
o  I -Level  test  equipment  requirements 

-  Manual  test  equipment 

-  Automatic  test  equipment  and  test  program  sets 

o  1-Level  training  requirements  to  support  skills  required 

-  On-the-job  training 

-  Formal  school  training 

o  1-Level  data  acquisition,  collection,  management,  analysis,  processing 
system  requirements 

o  Requirement  to  provide  1-Level  handshake  to  Depot-Level  diagnostic 
elements  (vertical  testability) 

DEPOT  MAINTENANCE  LEVEL: 

o  Define  FD/FI  functions  to  satisfy  Depot-Level  maintenance  operations, 
based  on: 
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-  Level-of-repair  analysis 

-  Sustainability  of  spares  pipeline 

-  Manpower  availability 

-  Skill  levels  available/required 

-  Repair  times 

-  Sparing  concepts 

-  Standardization  requirements/goals 

o  D-Level  technical  information  requirements  (including  maintenance  aids) 

o  D-Level  test  equipment  requirements 

-  Manual  test  equipment 

-  Automatic  test  equipment  and  test  program  sets 

o  D-Level  training  requirements  to  support  skills  required 

-  On-the-job  training 

-  Formal  school  training 

o  D-Level  maintenance  data  acquisition,  collection,  analysis,  processing 

o  Requirement  to  capture  and  utilize  factory  test  resources  and  results  and/or 

data  for  Depot  use  (vertical  testability,  vertical  diagnostics). 

Since  the  overall  diagnostic  capability  must  be  defined,  quantified,  designed, 
evaluated,  etc.,  it  is  best  defined  as  a  "diagnostic  subsystem.”  This  subsystem  can  be 
broken  down  into  its  component  parts  and  defined  in  a  type  of  format.  This  format  will 
facilitate  the  hierarchical  allocation  and  diagnostic  mix  optimization  process  because 
function  and  cost  parameters  can  be  quantitatively  assigned  to  each  element.  Alternative 
diagnostic  subsystems  may  then  be  easily  synthesized  and  evaluated. 
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CHECKLIST 

□f  Has  the  inherent  diagnostic  capability  of  the  prime 
system  architecture  been  included  in  the  analysis? 

Have  the  requirements  been  generated  for  both 
embedded  and  external  diagnostics? 

E*  Have  the  mission  and  operational  data  been  utilized 
in  the  diagnostic  requirements  generation? 
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WEAPON 
SYSTEM 
ACQ.  PHASE 

OPER. 

REQMTS. 

CONCEPT 

EXPLOR. 

DEM/ 

VAL 

FSD 

PROD/ 

DEPL. 

WEAPON 

SYSTEM 

ACTIVITIES 

ZX ZX 

CONTRACT 

AWARD 

DIAGNOSTIC 

ACTIVITIES 

ZX  ZX 

ALLOCATION  UPDATE 

DIAGNOSTIC  ACTIVITY 


Allocation  of  diagnostic  requirements  from  the  system  level  to  the  subsystem, 
and  unit  level  is  required  in  order  to  assign  specification  values  to  each  configuration  Item 
which  forms  part  of  the  weapon  system.  The  allocation  process,  which  is  normally  done 
by  the  contractor,  shall  assure  that  the  weapon  system  diagnostic  requirements  and  the 
constraints  on  the  diagnostic  subsystem  are  not  violated  during  the  "flow  down"  process. 

PROCEDURE 


Initial  allocation  of  diagnostic  requirements  to  lower  system  levels  must  be 
based  upon  the  time  demands  placed  upon  the  system  configuration  by  the  mission 
requirements. 

After  the  initial  set  of  diagnostic  requirements  has  been  defined,  a  diagnostic 
mix  is  postulated  from  the  synthesized  diagnostic  subsystem  alternatives  in  order  to 
implement  the  initial  set  of  diagnostic  requirements. 

Whereas  the  initial  diagnostic  requirements  are  driven  by  mission  time 
demands,  the  optimization  of  the  diagnostic  element  mix  is  driven  by  resource  constraints. 
Simply  stated,  the  requirements  generation  process  indicates  what  is  needed  and  the 
diagnostic  mix  generation  process  indicates  the  most  affordable  solutions.  A  risk  analysis 
performed  during  the  subsequent  phases  of  system  development  confirm  the  solutions  as 
feasible.  It  is,  therefore,  important  to  note  that  the  allocation  procedure  is  a  partial  step  in 
the  development  of  a  diagnostic  system.  During  the  diagnostic  element  optimization  and 
design  process,  it  may  be  cost  effective  to  reallocate  the  diagnostic  requirements  in  order 
to  achieve  better  implementations  with  respect  to  resource  constraints.  Many  of  these 
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tradeoffs  are  driven  by  both  technology  and  the  acquisition  business  decisions  that  are 
made  for  each  weapon  system  program.  For  example,  allocation  of  a  testability  strategy 
to  each  subsystem  may  not  be  feasible  due  to  the  existence  of  many  government- 
furnished  equipments  within  a  particular  weapon  system.  In  those  cases,  a  centralized 
system-level  test  approach  may  be  more  desirable.  A  shift  in  allocation  from  subsystem  to 
system  level  will  prove  effective  in  the  implementation  of  that  particular  weapon  system 
diagnostics. 

To  achieve  this  flexibility,  the  allocation  process  must  be  tied  to  the  system-level 
reliability  and  maintainablity  models.  This  model  will  contain  the  allocated  parameters  with 
traceability  back  to  system-level  parameters.  In  this  way,  as  the  program  proceeds  from 
Concept  through  Dem/Val  into  Full-  Scale  Development,  each  of  the  values  can  be  traded 
off  as  the  diagnostic  subsystem  is  configured  and  optimized  as  a  result  of  knowledge 
gained  from  trade  studies. 

GUIDANCE 


A  preliminary  diagnostic  allocation  should  be  prepared.  The  allocation  should 
include  all  diagnostic  elements  and  consideration  of  all  maintenance  levels.  The  allocation 
of  diagnostic  goals/values  should  be  accomplished  through  the  application  of  structured 
processes,  based  on  task  description  and  guidance  provided  within  applicable  military 
standards.  The  tasks  and  guidance  paragraphs  that  define  the  allocation  process  to  be 
employed  are: 


MIL-STD-499 

Task  10.2.3 

♦  Allocation 

MIL-STD-785 

Task  202 

Reliability  Allocation 

MIL-STD-470 

Task  202 

Maintainability  Allocation 

MIL-STD-2165 

Task  201 

Testability  Requirements. 

MIL-STD-499  addresses  the  entire  allocation  process  for  all  performance  and 
design  requirements.  Time  requirements,  which  are  prerequisites  for  a  function,  or  set  of 
functions,  affecting  mission  success,  safety,  and  availability  are  derived.  It  Is  essential  that 
the  diagnostic  requirements  be  derived  in  conjunction  with  the  entire  weapon  system 
allocation  process.  Reliability  and  maintainability  allocations  are  derived  as  part  of  the 
overall  weapon  system  allocations.  Thus,  they  have  a  direct  affect  on  the  diagnostic 
allocations.  Failure  rates  and  repair  rates  are  the  drivers  in  establishing  diagnostic 
allocations.  However,  other  considerations  dealing  with  safety  monitoring,  readiness 
monitoring,  and  logistic  functions  all  play  a  part  in  this  process.  The  allocation  of 
diagnostic  requirements  is  usually  performed  as  part  of  the  overall  LSA  process.  Closely 
tied  to  the  LSA  process  is  the  establishment  of  testability  requirements,  including 
performance  monitoring,  BIT,  test  equipment,  diagnostic  test  points,  etc. 
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It  is  important  that  this  allocation  process  includes: 

o  FD/FI  coverage  for  all  (1 00%)  faults  known  or  expected  to  occur  at  each 
maintenance  level,  and 

o  Quantification  of  all  diagnostic  elements. 

Figure  3  is  a  Notional  Diagnostic  Allocation  Specification,  which  exemplifies 
these  concepts.  This  allocation  process  is  also  closely  tied  to  the  optimization  process 
(Requirement  #2.3).  It  is  important  that  this  allocation  process  includes  quantification  of  all 
diagnostic  elements.  For  instance,  the  time  to  access  technical  information  can  determine 
whether  paper  or  electronic  delivery  of  technical  information  is  required.  Formal  training 
time  may  influence  the  need  for  on-the-job  training  aids. 

This  system-level  allocation  forms  the  basis  for  the  System  Specification 
discussed  under  Requirement  #1 .2.  It  also  is  followed  by  allocation  down  to  subsystem 
and  item  levels. 
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Allocate  Requirements  to  Item  Development  Specification 

System-level  diagnostic  requirements  are  allocated  down  to  subsystem  and  item 
levels  for  the  purpose  of  the  development  of  those  items.  Diagnostic  requirements  for 
Configuration  Items  (Cl)  support  two  distinct  requirements:  system  test  (primarily  BIT)  and 
shop  test  (ATE  and  GPETE). 

Quantitative  testability  requirements  for  each  Configuration  Item  are  allocated 
from  system  diagnostic  requirements  based  upon  FMEA  data,  relative  failure  rates  of  CIs, 
mission  criticality  of  the  CIs,  what  is  achievable  for  each  Cl  or  other  specified  criteria.  The 
failure  detection  level  of  the  Cl  is  weighted  by  the  items’  failure  rate  to  ensure  that  system- 
level  fault  detection  capability  is  achieved.  Table  3  is  an  example  of  an  allocation  of  a 
system-  level  BIT  fault  detection  requirement  which  is  allocated  to  five  configuration  items. 
The  table  shows  three  alternative  FD  allocations  which  meet  the  system-  level  BIT  FD 
requirement  of  95%. 

TABLE  3.  SAMPLE  ALLOCATION  OF  95%  BIT  FD  REQUIREMENT 


CON  FIGURATION 
ITEM 

FD  ALLOCATION 
#1 

FD  ALLOCATION 
#2 

FD  ALLOCATION 

#3 

A 

100 

.95 

.98 

.95 

B 

10 

.95 

.80 

1.00 

C 

50 

.95 

.70 

.98 

D 

200 

.95 

.99 

.90 

E 

100 

.95 

.98 

.99 

SYSTEM 

460 

.95 

.95 

.95 

The  BIT  performance  capability  and  testability  characteristics  of  GFE  portions  of 
the  system  should  be  considered  in  the  allocation.  For  example,  assume  a  GFE  item  has 
only  70%  BIT  fault-detection  capability.  In  order  to  accomplish  the  95%  system-level 
capability  required  in  the  above  example,  the  allocation  distribution  must  take  into  account 
the  capability  of  each  of  the  items  which  make  up,  or  contribute  to,  the  system  level.  The 
capability  of  the  GFE  then  serves  as  a  constraint  in  the  allocation.  In  the  above  example, 
given  that  Item  C  is  GFE  with  70%  BIT  fault-detection  capability,  the  FD  allocation  scheme 
#2  is  a  real  world  alternative  and  the  others,  #1  and  #3  are  not. 

Shop  test  requirements  are  determined  by  how  the  Cl  is  further  partitioned,  if  at 
all,  into  Units  Under  Test  (UUT).  Diagnostic  requirements  for  each  UUT  should  be 
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included  in  the  appropriate  Cl  Development  Specification.  These  parameters  are  not 
allocated  from  the  system-level  requirements,  but  rather  are  driven  by  the  diagnostic 
concept  of  off-line  test  requirements  of  the  Configuration  Items. 

In  many  digital  systems,  built-in  test  is  implemented  in  whole  or  in  part  through 
software.  Here,  diagnostic  requirements  will  appear  in  a  Computer  Program  Configuration 
Item  (CPCI)  development  specification.  The  CPCI  may  be  dedicated  to  the  built-in  test 
function  (i.e.,  a  maintenance  program)  or  may  be  a  mission  program  which  contains  test 
functions. 


2-22 


ALLOCATION  OF  DIAGNOSTIC  REQUIREMENTS 


REQUIREMENT  #2.2 


CHECKLIST 

□f  Were  the  system  reliability  and  maintainability  models 
used  in  the  diagnostic  allocation  process? 

□f  Are  the  allocated  values  traceable  to 
Weapon  System  Requirements? 

O'  Were  constraints  allocated  to  all  diagnostic  elements? 
Were  constraints  assigned  to  all  maintenance  echelons? 
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WEAPON 
SYSTEM 
ACQ.  PHASE 


OPER.  CONCEPT  DEM/ 

REQMTS.  EXPLOR.  VAL 


PROD/ 

DEPL. 


WEAPON 

SYSTEM 

ACTIVITIES 


CONTRACT 

AWARD 


PRELIMINARY 

DESIGN 


DIAGNOSTIC 

ACTIVITIES 


OPTIMIZE 

UPDATE 

UPDATE 

DIAG.  MIX 

OPTIMIZE  MIX 

DIAGNOSTIC  ACTIVITY 

Given  the  allocation  of  diagnostic  requirements  to  the  subsystem  and  unit  level, 
the  "diagnostic  subsystem"  must  be  defined  by  the  contractor  as  part  of  the  overall 
weapon  system  specification.  The  resulting  diagnostic  subsystem  includes  both 
embedded  and  external  support.  External  support  must  be  defined  at  all  levels  of 
maintenance  and  includes  technical  information,  support  equipment,  and  personnel 
numbers  and  skill  levels. 

PROCEDURE 

The  starting  point  for  developing  the  diagnostic  subsystem  is  the  generation  of  a 
diagnostic  subsystem  profile  from  the  weapon  system  characteristics  and  priorities.  Each 
of  the  diagnostic  elements  will  have  a  differing  impact  on  the  weapon  system 
characteristics.  For  example,  a  high  priority  constraint  on  logistic  support  would  favor  a 
high  degree  of  embedded  diagnostics.  On  the  other  hand,  constraints  on  personnel  may 
favor  technical  information  systems  with  a  high  degree  of  artificial  intelligence.  Operational 
constraints,  which  are  common  across  the  military  services,  are: 

o  Environmental  conditions  (temperature,  rain,  dirt,  salt  spray,  etc.) 

o  Operating  locations  (dispersed  vs.  centralized) 
(remote/accessible/inaccessible) 

o  Space  limitations  (for  personnel  and/or  test  equipment) 
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o  Mobile  or  fixed  maintenance  facilities 
o  Independent  operation  or  part  of  a  battle  group 
o  Manpower  constraints  (number  and  skill  levels). 

Analysis  of  the  weapon  system  characteristics  in  terms  of  their  impact  on  the 
support  elements  will  generate  various  support  element  diagnostic  profiles. 

The  diagnostic  profiles  will  portray  various  mixes  of  diagnostic  elements  for 
different  weapon  system  characteristics  and  constraints.  Each  of  ths  diagnostic  element 
profiles  infers  a  diagnostic  subsystem  which  can  be  built  and  delivered  with  the  weapon 
system.  The  optimization  issue  is  the  selection  of  a  diagnostic  subsystem  which  can  be 
implemented  at  low  risk  and  which  meets  the  requirements  allocated  to  system, 
subsystem,  and  unit  level. 

The  key  to  optimization,  therefore,  is  the  development  or  synthesis  of  various 
alternative  diagnostic  subsystems  based  upon  the  weighted  diagnostic  element  profiles. 
This  is  an  engineering  task  and  represents  an  important  aspect  in  the  overall  development 
of  a  diagnostic  capability  for  the  weapon  system.  By  generation  of  a  diagnostic 
subsystem,  early  in  Concept  Exploration,  the  overall  design  integration  of  support  and 
prime  design  elements  will  be  achieved.  During  the  Dem/Val  and  Full-Scale  Development 
Phases,  the  diagnostic  subsystem  is  refined  based  upon  trade  studies. 


The  key  is  to  identify  the  sensitivity  of  the  various  diagnostic  element  function 
contributions  to  the  overall  life  cycle  costs,  and  to  ensure  that  all  diagnostic  functional 
requirements  are  considered  and  included  as  part  of  the  total  diagnostic  subsystem 
synthesis. 

Each  diagnostic  subsystem  alternative  synthesized  is  evaluated  with  respect  to: 

o  Impact  on  Mission  Performance  Over  Time 

o  Impact  on  Resource  Requirements 

-  Acquisition  Cost 

-  Life  Cycle  Cost 

-  Manpower  Requirements 

o  Responsiveness  to  operational  constraints. 

The  evaluation  is  performed  by  assigning  values  related  to  the  evaluation 
factors  listed  above  to  the  diagnostic  subsystem  or  to  the  elements  of  the  diagnostic 
subsystem. 
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To  evaluate  the  mission  responsiveness  of  the  diagnostic  subsystem,  time 
demands  on  the  specific  weapon  system  configuration  must  be  characterized, 
performance  (interval  reliability)  for  each  mission  duration  is  calculated. 

To  evaluate  the  responsiveness  of  the  diagnostic  subsystem  to  operational 
constraints,  the  operational  constraints  must  be  assigned  qualitative  or  quantitative  values. 
The  impact  of  the  diagnostic  subsystem  characteristics  on  those  values  (time  demands) 
must  then  be  determined.  This  analysis  includes  availability  parameters  as  well  as 
mission  reliability  calculations  based  upon  the  stated  time  demands  and  subsystem 
utilization.  The  system  reliability  model  is  a  very  effective  and  available  tool  for  this 
analysis. 


To  evaluate  the  impact  of  the  diagnostic  subsystem  on  resources,  cost  factors 
must  be  assigned  to  each  element  of  the  diagnostic  subsystem.  Non-recurring 
(development)  and  recurring  (production  and  support)  costs  must  be  considered.  The 
manpower  requirements  associated  with  the  alternative  diagnostic  subsystems  must  be 
evaluated.  Existing  LCC  models  should  be  used  in  this  analysis.  Data  items  should  be 
standardized  wherever  possible. 

The  cost  deltas  associated  with  each  alternative  must  be  evaluated  with  respect 
to  the  off-line  maintenance  workload  costs  and  efficiencies  generated  by  the  alternative 
embedded  diagnostic  subsystems.  A  key  diagnostic  element  workload  driver  is  ambiguity 
group  size  and  RTOK  rates. 

Based  upon  the  evaluations  performed,  the  optimum  diagnostic  subsystem 
alternative  is  selected  and  the  weapon  system  diagnostic  concept  is  established  and 
documented.  The  diagnostic  concept  includes  prime  system  architecture  considerations, 
BIT  requirements  at  the  system  and  subsystem  levels,  test  equipment,  technical 
information  and  personnel  and  training  requirements  for  each  level  of  maintenance.  The 
diagnostic  function  of  each  element  must  be  clearly  and  quantitatively  defined  as  a 
diagnostic  requirement. 

Utilizing  the  above  procedure,  the  result  of  the  optimization  process  is  the 
development  of  a  diagnostic  subsystem  early  in  Concept  Exploration.  This  parallels  the 
development  effort  for  radar  subsystems,  fire  control  subsystems,  etc.  The  diagnostic 
subsystem  becomes  a  weapon  system  attribute  early  in  Concept  Exploration  and 
continues  to  evolve  during  subsequent  program  phases. 

GUIDANCE 


As  of  the  publication  date  of  this  document,  there  is  no  formal  guidance 
available  for  the  synthesis  and  optimization  of  a  diagnostic  subsystem.  Methodology  for 
performing  diagnostic  optimization  will  be  a  product  of  the  RADC  Automated  Testability 
Decision  Tools  Program  which  will  be  completed  and  published  in  mid-1990.  A  generic 
hierarchical  view  of  a  diagnostic  subsystem  which  includes  engineering  and  program 
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management  disciplines  as  well  as  embedded  and  external  support  elements  is  included 
below  to  serve  as  guidance  for  the  contractors.  This  indentured  diagnostic  subsystem 
breakdown  will  allow  costing  by  the  contractor  for  various  alternatives  proposed  to  satisfy 
the  diagnostic  requirements  which  have  been  allocated  at  all  system  levels.  As 
experience  data  is  accumulated  on  diagnostic  subsystem  effectiveness  and  cost,  it  will  be 
possible  to  predict  many  of  these  values  early  in  Concept  Exploration  using  the  diagnostic 
profile. 


DIAGNOSTIC  SUBSYSTEM  HIERARCHY 

I.  PROGRAM  MANAGEMENT/ENGINEERING 

A.  Requirements  Analysis 

B.  Diagnostic  Design  &  Analysis/Assessment 

C.  System  integration  &  Test 

D.  Maturation  Program 

II.  EMBEDDED  DIAGNOSTIC  ELEMENTS 

A.  System-Level  Diagnostic  Elements 

1 .  System-Level  Diagnostic  Hardware 

a.  Test  and  Maintenance  Bus 

b.  Sensors/Monitors 

c.  Diagnostic  Panei/Display 

d.  Embedded  ATE 

2.  System-Level  Diagnostic  Software 

a.  Status  Monitoring 

b.  Seif  Test/Expert  Systems 

c.  Prognostics 

d.  Reconfigurability 

3.  Diagnostics  Data  Collection  System 

B.  Subsystem  Diagnostics 

1 .  Subsystem  "A"  BIT 

a.  BIT  Hardware 

1 .  On  Chip 

2.  On  Printed  Circuit  Board 

b.  BIT  Software  &  Firmware 

c.  Interface  to  T&M  Bus 

2.  Subsystem  "B"  BIT  (Radar)  etc. 
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III.  EXTERNAL  DIAGNOSTIC  ELEMENTS 

A.  O-Level  Diagnostics 

1.  Technical  Information 

a.  Maintenance  Aids 

b.  Paper-Based  Manuals 

c.  Diagnostic  Data  Base 

2.  Test  Equipment 

a.  Manual  Test  Equipment 

b.  Automatic  Test  Equipment 

1 .  ATE  Hardware 

2.  Diagnostic  Software 

a.  Expert  Systems 

b.  Test  Program  Sets  (TPS) 

3.  ATE/ILS 

3.  Trained  Personnel 

a.  Manpower 

b.  Skills 

1.  Formal  Training 

2.  On-The-Job  Training 

4.  Diagnostic  Data  Collection/Analysis  System 

B.  1-Level  Diagnostics 

1.  Technical  Information 

a.  Maintenance  Aids 

b.  Paper-Based  Manuals 

c.  Diagnostic  Data  Base 

2.  Test  Equipment 

a.  Manual  Test  Equipment 

b.  Automatic  Test  Equipment 

1 .  ATE  Hardware 

2.  TPS 

3.  ATE/ILS 

3.  Trained  Personnel 

a.  Manpower 

b.  Skills 

1.  Formal  Training 

2.  On-The-Job  Training 
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4.  Diagnostic  Data  Collection/Analysis  System 
C.  D-Level  Diagnostics 

1.  Technical  Information 

a.  Maintenance  Aids 

b.  Paper-Based  Manuals 

c.  Diagnostic  Data  Base 

2.  Test  Equipment 

a.  Manual  Test  Equipment 

b.  Automatic  Test  Equipment 

1 .  ATE  Hardware 

2.  TPS 

3.  ATE/ILS 

3.  Trained  Personnel 

a.  Manpower 

b.  Skills 

1.  Formal  Training 

2.  On-The-Job  Training 

4.  Diagnostic  Data  Collection/Analysis  System 
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CHECKLIST 


O'  Have  alternative  "diagnostic  subsystems"  been 
generated  for  optimization  purposes? 

o'  Does  the  "diagnostic  subsystem"  include  all  maintenance 
levels  as  well  as  program  management  and  engineering? 


Was  a  life  cycle  cost  model  used  that  was  sensitive 
to  diagnostic  parameters? 
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WEAPON 
SYSTEM 
ACQ.  PHASE 


WEAPON 

SYSTEM 

ACTIVITIES 


OPER.  CONCEPT  DEM/ 

REQMTS.  EXPLOR.  VAL 


CONTRACT 

AWARD 


PROD/ 

DEPL. 


DIAGNOSTIC 

ACTIVITIES 


RISK 

ANALYSIS 


UPDATE 


DIAGNOSTIC  ACTIVITY 

The  initial  diagnostic  subsystem,  generated  to  implement  the  allocated 
diagnostic  requirements,  must  go  through  a  thorough  risk  analysis  during  the  Dem/Val 
Phase.  During  subsequent  Full-Scale  Development,  the  diagnostic  subsystem  is 
optimized  utilizing  results  of  trade  studies.  The  initial  diagnostic  element  mix  postulated 
during  Concept  Exploration  is  analyzed  by  the  contractor  for  risk  during  that  phase  by 
technology  assessments.  However,  risk  assessment  can  take  into  account  threat, 
technology,  resources,  schedule,  and  cost. 

PROCEDURE 

The  procedure  for  performing  risk  analysis  on  the  diagnostic  subsystem  will 
follow  the  same  type  of  assessments  conducted  for  risk  analysis  for  other  weapon  system 
elements.  For  example,  the  risk  assessment  for  a  radar  durirg  Lem/Val  will  include 
prototyping  of  its  new  development  components,  assessment  of  sci  v  «ule,  cost  risks,  and 
assessment  of  the  overall  technologies  involved  in  the  development  and  integration  of  a 
total  system  to  meet  the  performance  requirements.  Since  the  diagnostic  subsystem  will 
be  treated  as  a  major  element  of  a  weapon  system,  the  same  procedures  should  apply  for 
it.  Heretofore,  diagnostic  subsystems  were  not  treated  as  an  entity  and  risk  analysis  was 
limited  only  to  the  physical  diagnostic  hardware,  such  as  automatic  test  equipment  and 
built-in  test. 

Risk  assessment  shall  include  the  isolation  within  the  diagnostic  subsystem  of 
all  development  and  non-development  items.  For  development  items,  weighting  factors  in 
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terms  of  criticality  of  that  item  shall  be  assigned  and  the  items  shall  be  categorized  with 
respect  to  risk.  For  items  considered  high  development  risk,  workarounds  shall  be 
developed  and  trigger  points  for  decisions  on  their  implementation  shall  be  listed.  The  risk 
analysis  documentation  shall  be  utilized  to  impact  the  Statement  of  Work  for  the  Dem/Val 
Phase.  During  Dem/Val  high-risk  items  shall  be  prototyped  and  demonstrated  to  the 
satisfaction  of  the  Government  Program  Manager. 

GUIDANCE 


The  Defense  Systems  Management  College  has  generated  guidance  on  risk 
management,  which  includes  risk  assessment,  risk  analysis,  risk  handling  techniques,  and 
risk  control.  This  guidance  covers  risk  management  for  the  entire  weapon  system,  but  is 
equally  relevant  to  the  weapon  system’s  diagnostic  capability.  Both  risk  assessment  and 
risk  analysis  need  to  be  addressed  early  in  the  development  of  the  weapon  system.  Risk 
assessment  is  the  process  of  examining  a  situation  and  identifying  areas  of  potential  risk. 
Risk  analysis  is  examining  the  change  of  consequences  with  the  modification  of  risk  input 
variables.  At  the  time  this  Contractor  Program  Managers  Guide  was  issued,  the  Defense 
Systems  Management  College  is  publishing  a  risk  management  guide,  which  further 
defines  the  methodology  for  doing  risk  assessment  and  analysis. 

MIL-STD-1 388-1  (Logistics  Support  Analysis)  contains  in  Tasks  203,  205,  and 
303  guidance  on  comparative  analysis,  supportability  related  design  factors,  and 
evaluation  of  alternatives  for  trade-off  analysis,  all  of  which  are  directly  related  to  the 
weapon  system’s  diagnostic  capability. 

Lessons  learned  have  pointed  to  some  overriding  areas  of  risk  which  must  be 
considered  during  the  initial  risk  assessment  and  analysis.  These  high-risk  areas  are 
listed  in  the  following  paragraphs: 

1 .  The  logistic  support  analysis  process  will  usually  generate  requirements  for 
each  of  the  logistic  elements  comprising  the  overall  logistic  system.  These  requirements 
are  based  upon  inputs  regarding  the  level  of  embedded  support  to  be  designed  into  the 
weapon  system.  The  Logistic  Element  Manager,  given  these  inputs,  proceeds  to  develop 
sparing  requirements,  support  equipment  requirements,  training  requirements,  etc.  A 
large  program  risk  area  occurs  when  the  promised  embedded  support  area  does  not 
materialize.  It  Is  imperative,  therefore,  to  close  the  loop  between  assessment  during 
Dem/Val  of  the  diagnostic  element  capability  and  that  impact  on  all  logistic  elements. 

2.  A  second  major  risk  area  occurs  when  a  prime  weapon  system,  which  has 
been  developed  for  a  specific  maintenance  strategy  and  concept,  is  utilized  in  a 
completely  different  mission  environment.  For  example,  a  major  weapon  system  deployed 
in  a  three-level  maintenance  environment  may  be  required  to  operate  for  extended  periods 
of  time  in  a  dispersed  operating  location  with  less  than  full  support.  The  sustainability  and 
mobility  requirements  imposed  upon  that  weapon  system  may  not  have  been  included  with 
sufficient  priority  in  the  initial  analysis  to  develop  capability  for  that  operational 
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environment.  It  is,  therefore,  imperative  that  as  part  of  the  risk  analysis,  the  assessment  of 
weapon  system  characteristics  and  the  application  of  weapon  system  priorities  be 
reviewed  prior  to  commitment  of  system  development  resources. 

3.  A  third  high  risk  area  worthy  of  special  consideration  is  the  analysis  of  the 
very  large  scale  integrated  circuits  and  very  high  speed  integrated  circuits  (VLSI/VHSIC). 
Despite  the  intensive  use  of  on-chip  testing  for  these  devices,  it  is  imperative  that  a 
standard  systems  approach  be  generated  by  the  contractor.  Testability  techniques 
including  signature  analysis  and  boundary  scan  designs  must  be  evaluated  at  the  system 
and  subsystem  level  prior  to  commitment  of  development  resources.  Standardization  by 
the  contractor  of  the  embedded  support  architecture  will  eliminate  many  high-risk  problems 
caused  by  multiple  vendors  supplying  different  types  of  on-chip  testing. 

4.  A  fourth  high  risk  area  occurs  when  weapon  system  managers  fail  to 
comprehend  and  implement  the  existing  fielded  maintenance  standards  that  are  used  to 
support  the  deployed  system.  For  example,  the  military  has  for  many  years  been 
formalizing  the  use  of  IEEE-STD  716  C/ATLAS  language  for  Depot  maintenance.  The 
CASS,  IFTE  and  MATE  programs  have  institutionalized  this  approach.  Despite  this  level 
of  standardization,  many  programs  completely  ignore  this  fact  during  the  Dem/Val  and 
FSD  Phases  of  a  program.  Since  the  targeted  Depot  ATE  has  been  standardized,  it  is 
possible  to  develop  test  programs  starting  with  Factory-level  testing  through  integration 
and  test  of  the  products  that  are  compatible  and  easily  translatable  to  the  fielded 
environment.  This  concept,  called  vertical  commonality,  will  mature  the  test  programs  so 
that  during  deployment  the  logistic  system  will  have  a  major  capability  and  remove  many 
of  the  risks  associated  with  transition  from  interim  contractor  support  to  full  government 
support.  Utilizing  expert  system  knowledge  during  these  same  phases  will  allow  the  test 
program  to  contain  levels  of  artificial  intelligence  to  extract  and  utilize  experience  data  on 
prior  failures  during  the  Deployment  Phase. 

5.  The  fifth  high  risk  area  is  the  incorporation  of  government  furnished 
equipment  (QFE)  in  weapon  systems.  Care  must  be  taken  to  ensure  that  the  diagnostic 
requirements  and  capability  are  known  and  verified.  The  Government  Program  Manager 
must  be  informed  if  the  required  weapon  system  diagnostic  capability  is  compromised  by 
deficiencies  in  the  GFE. 

6.  The  sixth  and  final  large  risk  area  is  the  integration  and  test  of  the  weapon 
system  prior  to  delivery.  Since  weapon  systems  have  become  extremely  software 
dependent  and  since  many  weapon  systems  are  multi-mission  in  nature  utilizing  shared 
resources,  it  is  imperative  that  the  integration  and  test  function  in  a  program  be  utilized  to 
remove  as  much  risk  as  possible  from  the  weapon  system.  Integration  of  the  diagnostic 
elements  into  the  weapon  system  will  provide  a  major  "handle”  for  the  contractor  in  terms 
of  enhancing  the  integration  and  test  functions.  If  no  attention  is  paid  early  in  the  game  to 
this  high  potential  risk,  the  integration  and  test  functions  will  be  extremely  time  consuming, 
may  not  come  together  on  schedule,  and  may  cause  program  hardships.  If  properly 
achieved,  integration  and  test  can  be  streamlined  to  recover  much  of  the  upfront  monies 
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spent  on  enhanced  testability  features.  It  is  therefore  imperative  that  this  area  be  given 
serious  attention  by  risk  assessment  studies  early  in  Concept  Exploration  and  again  in 
Dem/Val  and  Full-  Scale  Development. 
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CHECKLIST 

O'  Was  risk  analysis  performed  for  the  entire 
"diagnostic  subsystem"? 


O' 


During  DEM/VAL  were  testing/prototyping  efforts 
undertaken  to  determine  feasibility  or  achieving 
diagnostic  performance  requirements? 


Were  adjustments  planned  for  in  those  cases  where 
one  of  the  diagnostic  elements  fails  to  meet 
expectations? 

O'  Were  the  weapon  system  priorities  taken  seriously? 
□f  Have  the  integration  and  test  risks  been  defined? 
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DESIGNING  THE  DIAGNOSTIC  CAPABILITY 
OVERVIEW 


The  design  of  the  diagnostic  capability  is 
fractionated  among  a  number  of  system 
engineering  and  supportability  functions. 
Reliability,  maintainability,  integrated  logistic 
support,  testability,  human  engineering,  and 
safety  considerations  all  play  significant  roles  in 
determining  the  requirements  of  the  diagnostic 
capability  and  the  design  of  this  capability.  The 
design  process  is  further  fractionated  by  the 
relegation  of  this  capability  to  the  various  levels  of 
maintenance.  The  diagnostic  design  process  is 
controlled  by  a  large  number  of  military  standards 
which  deal  with  the  design  process  and  criteria. 
All  of  these  "pieces"  of  the  design  process  must 
not  only  work  together,  but  the  diagnostic  data 
produced  must  be  available  at  specific  times.  A 
break  in  one  of  the  links  can  compromise  the 
design.  A  cohesive,  integrated  design  process  is 
required.  It  is  the  Program  Manager’s  job  to  assure 
that  this  integration  is  realized  and  the  designer’s 
job  to  produce  this  effective  diagnostic  capability  in 
an  efficient  manner. 


TESTABILITY/ 

DIAGNOSTICS 

in — 


PROGRAMMATIC 


REQUIREMENTS 


ASSESSMENT 


REVIEWS 


EVALUATION 


MATURATION 


IMPORTANT  CONSIDERATIONS  TO  BE  ADDRESSED 


Reqmt 

3.1  Assure  cohesiveness  and  efficiency  in  the  design  of  the  diagnostic 
capability. 

3.2  Establish  diagnostic  design  criteria  which  can  be  effectively  utilized. 
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WEAPON 
SYSTEM 
ACQ.  PHASE 


OPER.  CONCEPT  DEM/ 

REQMTS.  EXPLOR.  VAL 


FSD  PROD/ 
DEPL. 


WEAPON 

SYSTEM 

ACTIVITIES 


SDR  PDR  CDR 


DIAGNOSTIC 

ACTIVITIES 


DIAGNOSTIC  PRELIM.  DETAIL  FABRICATION 

SPEC.  DESIGN  DESIGN 


DIAGNOSTIC  ACTIVITY 

The  Contractor  Program  Management  function  is  to  ensure  that  an  effective  and 
efficient  diagnostic  design  process  is  instituted  and  implemented. 

PROCEDURE 

The  cohesiveness  of  the  diagnostic  design  process  is  dependent  upon  the 
cohesiveness  of  the  design  information  flow.  Many  factors  impact  the  effectiveness  and 
efficiency  of  this  information  flow.  The  first  is  timing  -  What  is  done  and  in  what  sequence 
is  it  done?  The  second  factor  relates  to  the  various  disciplines  involved  in  the  design  of 
the  diagnostic  capability.  These  disciplines  are  controlled  by  a  sizeable  number  of  military 
standards,  which  relate  to  reliability,  maintainability,  testability,  safety,  human  engineering, 
software,  and  training.  These  standards  tend  to  fractionalize  the  design  of  the  diagnostic 
capability,  inasmuch  as  each  plays  a  significant  role  in  the  process.  The  third  factor  deals 
with  the  automation  of  the  design  process.  Computer-aided  tools  can  promote  the 
cohesiveness  and  the  efficiency  of  the  process.  Thus,  the  Contractor  Program  Manager 
must  understand  the  interfaces  among  these  various  engineering  and  logistic  functions  to 
assure  that  the  necessary  cohesiveness  is  achieved,  in  addition,  the  automation  of  the 
diagnostic  design  process  should  be  supported  and  encouraged  to  provide  a  more 
efficient  process  and  a  more  effective  diagnostic  capability. 


3-3 


PROVIDING  A  COHESIVE  DIAGNOSTIC  DESIGN  PROCESS  REQUIREMENT  #3. 1 


GUIDANCE 


The  guidance  provided  in  this  section  is  designed  to  permit  maximum  visibility 
into  the  diagnostic  design  process.  The  Contractor  Program  Manager  must  understand 
the  design  process  flow,  timing,  and  data  requirements  which  must  be  satisfied.  In 
addition,  it  is  important  to  recognize  that  current  data  item  procurement  practices  may  not 
always  be  supportive  of  the  design  activity  in-process  data  needs.  Very  often,  the  CDRL 
and  associated  DID  do  not  adequately  reflect  these  in-process  needs.  The  high  data  item 
generation/revision  costs  generally  experienced  are  strong  motivators  for  delaying  data 
item  preparation  to  a  point  where  the  design  has  stabilized.  Such  motivation  is  in  direct 
conflict  with  the  utilization  of  the  data  to  make  design  decisions.  A  complete,  detailed  data 
submittal  indicating  that  the  design  is  flawed  is  of  little  use  after  the  design  has  been 
completed.  The  guidance  that  follows  has  been  designed  to  provide  the  necessary  insight 
into  the  design  process,  which  will  assist  the  Program  Manager  in  the  progress 
assessment  and  decision-making  process. 

Design  Environment 

The  diagnostic  design  environment  is  an  essential  component  of  the  overall 
diagnostic  design  activity,  which  has  been  established  by  the  contractor  in  response  to  the 
RFP  requirements.  This  environment  encompasses  both  the  implementation  methodology 
and  the  specialty  coordination  associated  with  the  diagnostic  design  process.  Evidence  of 
these  should  be  apparent  in  the  interim  products  of  the  design  effort,  which  are  made 
available  to  the  government  program  management  function  (at  both  informal  in-process 
reviews  and  formal  system-level  design  reviews). 

Diagnostic  design  is  characterized  by  its  iterative  nature  and  a  high  degree  of 
interdependence  with  the  supportability  engineering  specialties  (i.  e.,  reliability, 
maintainability,  integrated  logistic  support,  testability,  human  engineering,  and  safety). 
The  allocation  of  diagnostic  resources  must  be  based  on  inputs  from  these  disciplines. 
Therefore,  the  timing  and  quality  of  data  interchanges  must  be  in  accordance  with  the 
program  plans.  A  breakdown  in  data  availability  and  exchange  can  be  responsible  for 
program  delays  and  shortfalls  in  the  fielded  diagnostic  capability. 

The  data  flow  required  to  develop  the  composite  diagnostic  capability  must  be 
responsive  to  the  diagnostic  mix  established  for  the  specific  system  under  consideration. 
Embedded  diagnostic  features,  such  as  built-in  test  (BIT),  built-in  test  equipment  (BITE), 
system  integrated  test  (SIT),  performance  monitoring,  status  monitoring,  embedded 
training,  embedded  maintenance  aiding,  adaptive  Al-based  diagnostic  systems,  etc.,  are 
an  integral  part  of  the  prime  equipment  design.  Therefore,  the  diagnostic  data  flow 
associated  with  these  features  must  be  incremental  and  continue  until  the  detail  prime 
system  Configuration  Item  designs  are  complete.  For  the  external  diagnostic  elements, 
such  as  automatic  test  equipment  and  the  associated  test  program  sets,  manual  test 
equipment,  portable  maintenance  aids,  technical  information  (hard  copy  or  electronic 
delivery),  training,  etc.,  the  diagnostic  data  flows  into  the  LSA  process  up  to  the  point 
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where  the  firm  requirements  for  these  diagnostic  elements  can  be  established.  Once  firm 
requirements  exist,  the  diagnostic  design  environment  must  facilitate  a  smooth  transfer  of 
data,  which  is  sufficient  in  terms  of  detail  and  format  to  permit  fabrication  of  the  required 
external  diagnostic  capability. 

Program  management  must  develop  an  understanding  for  the  complexity  of  the 
data  flow  requirements  associated  with  the  program  under  consideration.  Given  the 
required  understanding,  maintaining  cognizance  over  the  content  and  timeliness  of  data 
availability  cannot  be  overemphasized. 

Table  4  is  a  listing  of  the  major  military  standards  which  influence  the  design  of 
the  diagnostic  capability.  Some  of  these  military  standards  are  programmatic  in  nature,  in 
that  they  establish  a  specific  program  and  described  the  tasks  which  can  be  undertaken. 
The  remainder  of  the  standards  are  process  or  product-oriented.  As  can  be  seen,  these 
various  standards  influence  various  aspects  in  the  design  of  the  diagnostic  capability, 
starting  from  establishing  diagnostic  requirements,  through  the  design  and  assessment  of 
the  diagnostic  capability.  There  is  a  sequence  of  tasks  and  procedures  cited  in  these 
standards  which  can  be  applied  to  the  diagnostic  capability.  The  interfaces  and 
relationships  between  these  various  activities  are  complex  and  cannot  be  easily 
diagrammed  to  promote  understanding.  Establishing  diagnostic  requirements  is  described 
in  Requirement  #2,  and  the  assessment  is  described  under  Requirement  #4.  Thus  the 
following  guidance  will  address  the  design  of  the  embedded  and  external  diagnostic 
capability. 
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TABLE  4.  MILITARY  STANDARDS  APPLICABLE  TO  THE  DESIGN  OF  THE  DIAGNOSTIC  CAPABILITY 


MIL-STD-471  MalnUJnabflRy 
Domonct  ration 


MIL-STO-756  RolabfIKy 
Modalng  &  Praid. 


MtL-STD-1379  Contract 
Training  Prog. 


PROVIDING  A  COHESIVE  DIAGNOSTIC  DESIGN  PROCESS  REQUIREMENT  #3.1 
Design  Integration 


Figure  4  is  a  simplified  diagram  of  the  information  flow  in  the  design  of  the 
diagnostic  capability.  The  design  process  begins  with  a  maintenance  concept  and  design 
data,  such  as  specifications,  block  diagrams  and  schematics.  Establishing  tne  system’s 
architecture  is  the  next  step.  System's  architecture  has  a  major  impact  on  the  design  of 
the  fielded  diagnostic  capability.  The  concept  of  fault  tolerance  supports  the  maintenance 
concept  by  promoting  graceful  degradation  of  the  system’s  performance,  thereby  allowing 
the  maintenance  to  be  performed  at  the  user’s  convenience  rather  than  dictated  by  when 
faults  occur.  Design  for  testability  concepts  play  an  important  part  at  this  time.  Partitioning 
especially  is  closely  tied  to  fault  tolerance,  because  the  performance  monitoring  capability 
must  be  able  to  detect  failed  items  in  order  that  the  capability  of  the  system  is  known,  that 
necessary  switching  to  alternate  means  is  facilitated,  and  that  maintenance  actions  can  be 
identified. 


The  Failure  Modes  and  Effects  Criticality  Analysis  (FMECA)  utilizes  the  system’s 
architecture  and  design  data  to  determine  the  modes,  causes  and  effects  of  item  failures. 
This  data  drives  the  maintenance  and  safety  requirements  which  in  turn  help  to  establish 
the  diagnostic  logic,  test  point  selection,  and  test  requirements.  From  this  information,  the 
diagnostic  capability  is  designed  and  fabricated,  including  the  testing,  (built-in  and 
external),  technical  information,  training,  and  personnel  capability.  Obviously  this  entire 
process  is  iterative  in  nature  -  a  factor  not  indicated  in  Figure  4. 

The  concept  of  vertical  testability  was  introduced  years  ago.  In  essence,  this 
concept  addressed  the  compatibility  of  testing  among  all  levels  of  maintenance,  including 
factory  testing.  The  core  of  this  concept  is  twofold.  The  first  is  the  establishment  of  a 
Cone  of  Tolerance  among  these  levels,  and  the  second  deals  with  the  compatibility  of 
environments  under  which  these  tests  are  performed. 

The  Cone  of  Tolerance  concept  is  depicted  in  Figure  5,  in  which  the  testing 
tolerances  are  widened  as  the  unit  is  tested  closer  to  Its  operational  environment. 
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TOLERANCE 


FIGURE  5.  CONE  OF  TOLERANCE 


The  compatibility  of  testing  environments  can  be  implemented  best  through  the 
use  of  common  test  equipment  at  Intermediate,  Depot,  and  Factory  Levels. 

Extension  of  this  vertical  testability  concept  is  recommended  for  the  entire 
fielded  diagnostic  capability.  Figure  6  depicts  this  concept,  in  which  vertical  testing  is 
shown  on  the  left  and  is  joined  by  technical  information  and  personnel  and  training 
compatibility  requirements.  Not  only  is  this  compatibility  required  vertically,  but  also 
horizontally.  All  elements  that  make  up  the  diagnostic  capability  must  be  compatible  at 
each  maintenance  level. 

This  concept  of  vertical  and  horizontal  compatibility  is  key  to  the  integration  of 
diagnostic  capability.  The  entire  process  is  driven  by  the  diagnostic  logic  which  effects  the 
requirements  for  all  of  the  diagnostic  elements.  This  diagnostic  logic  can  be  established 
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by  a  variety  of  means  including  the  use  of  maintenance  dependency  charts,  fault  trees, 
etc.  To  implement  this  concept,  a  series  of  matrices  similar  in  format  to  Figure  6  can  be 
prepared  at  various  system  hierarchy  levels  (e.g.,  system,  subsystem,  LRU,  SRU).  These 
matrices  should  be  tailored  to  the  unique  requirements  of  a  specific  weapon  system  and 
may  be  used  in  conjunction  with  other  required  data  deliverables  (e.g.,  test  requirements 
document). 


FIGURES.  Design  Integration  of  Diagnostic  Capability 

AUTOMATION  OF  THE  DIAGNOSTIC  DESIGN  PROCESS 

Automation  of  the  diagnostic  design  process  is  encouraged  to  provide  a  more 
efficient  and  effective  design  process.  The  diagnostic  design  process  should  be  an 
integral  part  of  prime  system  computer-aided  engineering  and  design. 

The  added  efficiency  and  effectiveness  in  the  use  of  automation  is  reflected  in  a 
number  of  ways.  The  effect  of  changes  in  either  the  diagnostic  design  or  the  prime  system 
design  can  be  readily  ascertained  as  the  design  progresses.  This  iterative  process  then 
can  give  the  Contractor  Program  Manager,  as  well  as  the  designer,  information  on 
whether  or  not  the  diagnostic  specification  requirements  will  be  met.  In  addition, 
automation  permits  the  concurrent  development  and  evaluation  of  the  entire  diagnostic 
capability  along  with  the  remainder  of  the  prime  system. 
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The  diagnostic  design  and  assessment  tools  enhance  the  effectiveness  and 
efficiency  of  the  process.  A  description  of  available  tools  and  processes  is  available  in  the 
Design  Encyclopedia  Guide. 
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CHECKLIST 


O'  Has  a  diagnostic  design  environment  been  adequately 
defined  and  imposed  to  ensure  that  diagnostic  design 
requirements  are  considered  as  part  of  the  mainstream 
design  effort? 

□r  Has  a  concerted  effort  been  made  to  assure  vertical 
and  horizontal  integration  and  compatibility  of 
all  elements  which  comprise  the  diagnostic  capability? 
Has  this  been  documented  for  review? 

Have  steps  been  taken  to  utilize  automation  of  the 
diagnostic  design  process  to  enhance  design  efficiency 
and  to  improve  the  effectiveness  of  the  fielded 
diagnostic  capability? 

□f  What  measures  will  be  taken  to  ensure  that  vertical  and 
horizontal  diagnostic  concepts  are  being  implemented? 

o'  How  are  the  interfaces  with  training,  technical  infor¬ 
mation,  and  off-line  test  being  managed  to  facilitate 
concurrent  delivery  of  weapon  system  and  support  to  the 
extent  required  for  test  and  evaluation  and  maturation? 
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WEAPON 
SYSTEM 
ACQ.  PHASE 

OPER. 

REQMTS. 

CONCEPT 

EXPLOR. 

DEM/ 

VAL 

FSD 

PROD/ 

DEPL. 

WEAPON 

SYSTEM 

ACTIVITY 

£  £  £  zs 

SYSTEM  SYS.  PREL.  DETAIL 

ANALYSES  SPEC.  DESIGN  DESIGN 

DIAGNOSTIC 

ACTIVITIES 

DIAGNOSTIC  DESIGN  fSs> 

DIAGNOSTIC  ACTIVITY 

Design  of  the  diagnostic  capability  and  the  elements  that  make  up  this  capability 
are  initiated  early  in  weapon  system  development.  It  begins  soon  after  initial  analyses  and 
allocation  are  completed  and  extends  at  least  until  the  end  of  Full-Scale  Development. 
Design  criteria  and  guidance  need  to  be  available  for  use  as  the  diagnostic  capability 
design  progresses.  Obviously,  the  bulk  of  this  design  guidance  is  utilized  by  the  designer 
of  the  prime  system  and  its  support  capability.  The  Contractor  Program  Manager  needs  to 
be  aware  of  the  type  of  guidance  that  is  available  and  if  the  contract  specifies  any  design 
criteria. 

PROCEDURE 

Design  criteria  and  guidance  relating  to  the  diagnostic  capability  and  individual 
diagnostic  elements  are  available  from  a  number  of  sources,  including  standards, 
handbooks,  and  guides.  Most  often,  this  guidance  is  not  a  contractual  requirement, 
except  when  a  specific  military  standard  is  invoked.  However,  for  the  most  part,  the 
contractor  should  utilize  this  guidance  material  as  he  sees  fit,  as  long  as  diagnostic 
requirements  are  met  and  interfaces  are  controlled. 

Guidance  to  the  Contractor  Program  Manager  consists  of  identifying  applicable 
guidance  documents,  and  where  published  material  is  not  readily  available,  limited 
guidance  is  furnished. 
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Of  particular  concern  to  the  Contractor  Program  Manager  is  the  implementation 
of  inherent  testability  in  his  system.  Design  of  a  cost  effective  and  efficient  diagnostic 
capability  must  start  with  foundation  of  inherently  testable  circuits  and  assemblies. 
Inherent  testability  is  based  soley  on  basic  design  features  such  as  physical  and  electrical 
partitioning,  controllability,  observability,  and  test  point  placement.  The  inherent  testability 
of  a  unit  will  greatly  reduce  the  cost,  complexity,  efficiency  and  development  time  of  both 
the  on-line  and  off-line  diagnostics  required.  Good  diagnostics  design  and  design 
practices  built  on  a  foundation  of  circuit  and  unit  designs,  which  are  inherently  testable,  will 
close  the  loop  and  provide  the  most  effective  system  diagnostics  capability  at  minimum 
cost. 


In  the  commercial  sector,  many  major  companies  have  made  corporate 
decisions  to  design  electronics  products  to  be  testable.  The  impetus  for  testability  in  the 
commercial  sector  is  based  upon  the  time  and  cost  savings  achieved  in  the  factory  test 
environment.  Factory  rework  time  and  costs  are  decreased  based  upon  confident,  early 
detection  and  diagnosis  of  faults.  Test  coverage,  test  time,  diagnostics  time,  fixturing  and 
programming  costs  are  all  significantly  impacted  by  the  inherent  testability  characteristics 
of  a  product.  Further  benefits  of  testability  are  achieved  in  the  field  maintenance 
environment.  The  design  for  testability  decision  in  the  commercial  sector  is  based  upon  a 
return  on  investment  parameter  where  the  beneficial  impacts  of  testability  are  evaluated 
against  the  costs  associated  with  testability  implementation  -  recurring  and  nonrecurring. 

In  the  military  environment,  the  key  driver  for  testability  is  the  impact  of  test 
effectiveness  on  mission  success  and  life  cycle  cost.  The  quantification  of  testability 
benefits  in  military  applications  has  been  difficult  to  assess.  The  impact  of  improved  test 
effectiveness  shown  in  the  chart  below  (Figure  7)  intuitively  leads  to  significant  benefits. 
However,  quantification  of  these  benefits  can  only  be  done  on  a  case-by-case  basis. 


FIGURE  7.  IMPACT  OF  IMPROVED  TEST  EFFECTIVENESS 
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GUIDANCE 

Detailed  design  criteria  and  guidance  in  relation  to  the  entire  diagnostic 
capability,  as  well  as  for  each  diagnostic  element,  are  addressed  in  detail  in  the  Design 
Encyclopedia  Guide.  A  substantial  amount  of  information  that  is  contained  in  that  guide  is 
not  available  in  other  existing  guides.  Guidance  contained  in  this  Contractor  Program 
Managers  Guide  is  limited  to  references  to  other  available  design  criteria  and  guidance. 

The  following  are  references  to  existing  design  criteria  and  guidance. 

General  Guidance 

1 .  MIL-STD-454,  Standard  General  Requirements  for  Electronic  Equipment 

This  standard  covers  the  common  requirements  to  be  used  in  military 
specifications  for  electronic  equipment.  Reliability,  maintainability,  and 
human  engineering  requirements  are  included  m  this  standard.  However,  for 
these  types  of  engineering  disciplines,  the  guidance  stresses  that  this 
standard  does  not  establish  requirements  and  must  not  be  referenced  in 
contractual  documents.  These  three  requirement  examples  offer  direction 
on  what  should  be  considered  in  preparing  contractual  documents. 

2.  MIL-STD-415,  Design  Criteria  for  Test  Provisions  for  Electronic  Systems  and 
Associated  Equipment 

This  standard  establishes  design  criteria  for  test  provisions  that  permit  the 
functional  and  static  parameters  of  electronic  systems  and  associated 
equipment  to  be  monitored,  evaluated,  or  isolated.  The  standard,  in  its 
present  form,  (Revision  D)  addresses  older  technologies  and  thus,  if 
referenced  in  contractual  documents,  must  be  tailored  to  address  only 
certain  provisions  in  this  standard. 

3.  The  RADC  Reliability  Engineers  Tool  Kit 

The  Tool  Kit  is  intended  for  use  by  a  practicing  reliability  and  maintainability 
(R&M)  engineer.  Emphasis  is  placed  on  his  role  in  the  various  R&M 
activities  of  an  electronic  systems  development  program.  The  Tool  Kit  is  a 
compendium  of  useful  R&M  reference  information  to  be  used  in  everyday 
practice. 

System  Architecture 

There  are  a  number  of  guides,  which  address  the  architecture  of  the  system 
design,  that  promote  improvements  in  the  system's  diagnostic  capability.  Included  are: 
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1.  Architecture  Specification  for  PAVE  PILLAR  Avionics,  January  1987, 
SPA90099001A 

This  specification  addresses  the  advanced  avionics  architecture  which  is 
specifically  targeted  for  advanced  tactical  fighters  and,  in  general,  for  all 
military  aircraft  applications.  This  architecture  promotes  a  much-improved 
approach,  which  will  foster  an  improved  diagnostic  capability.  An  example  of 
this  approach  is  contained  in  the  Design  Encyclopedia  Guide. 

2.  Reliability,  Testability  Design  Considerations  for  Fault  Tolerant  Systems 
(RADC-TR-84-57) 

Furnished  reliability  and  testability  evaluation  and  application  guidance  for 
fault-tolerant  designs. 

Testability 

There  are  a  number  of  guidance  documents  which  address  testability  issues. 
Some  of  these  are  listed  below.  These  deal  with  the  design  techniques  of  controllability, 
observability,  and  partitioning.  Controllability  is  a  design  attribute  which  describes  the 
extent  to  which  signals  of  interest  may  be  controlled  by  the  test  process.  It  relates  to 
difficulty  of  test  generation,  length  of  test  sequence,  and  diagnostic  resolution. 
Observability  is  another  design  attribute  which  describes  the  extent  to  which  signals  of 
interest  may  be  observed  by  the  test  process.  The  emphasis  is  upon  selection  of  the  most 
appropriate  test  points.  Partitioning  deals  with  both  the  physical  hardware  and  the 
functional  partitioning  of  the  circuitry.  Test  times  and  test  generation  costs  escalate  rapidly 
as  partitioning  size  increases. 

1.  RADC  Testability  Notebook,  Final  Technical  Report,  June  1982 

This  notebook  presents  a  consolidation  of  information  relating  to  testability 
design  techniques,  procedures,  cost  trade-off  tools,  and  the  relationship  of 
testability  to  other  design  disciplines  and  requirements.  Specific  examples  of 
good  testability  design  are  contained  in  this  document. 

2.  Avionics  Testability  Design  Guide  --  MATE  Guide  3,  Part  Three,  Testability 
Design  Handbook 

This  portion  of  the  MATE  Guides  discusses  testability  design  techniques. 

3.  MIL-STD-2165,  Testability  Program  for  Electronic  Systems  and  Equipments 

Appendix  B  of  MIL-STD-2165  cites  a  series  of  factors  which  affect  the 
inherent  testability  of  a  weapon  system.  This  information  can  be  used  either 
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as  design  guidance  or,  if  weighted  and  scored,  can  actually  provide  a  Figure 
of  Merit  for  a  specific  system/unit. 

4.  Testability  Analysis  Handbook  (Draft) 

At  the  time  of  printing  the  Contractor  Program  Managers  Guide,  the 
Testability  Analysis  Handbook  was  in  draft  form.  Publishing  is  scheduled 
during  FY89.  The  Preparing  Activity  is  the  Naval  Sea  Systems  Command, 
CEL-DST.  This  handbook  provides  a  systematic  methodology  for 
implementing  testability  analysis  and  design  requirements,  which  are 
prescribed  in  MIL-STD-2165,  Tasks  201, 202,  and  203. 


Built-In  Test 

1 .  Built-In  Test  Design  Guide-Joint  AMC/CNO/AFLC/AFSC  Commanders,  April 
1987 

This  Joint  Service  BIT  Design  Guide  provides  detailed  guidelines  on  the 
implementation  of  BIT,  including  BIT  design  techniques  at  all  levels  within 
the  weapon  system. 

2.  Smart  BIT  (RADC-TR-85-1 48) 

Application  of  Artificial  Intelligence  techniques  in  the  design  of  BIT,  to 
minimize  false  alarms,  retest  OKs  and  non-required  maintenance. 

3.  Sensor  Handbook  for  Test,  Monitoring,  Diagnostic,  and  Control  System 
Applications  to  Military  Vehicles  and  Machinery,  National  Bureau  of 
Standards 

This  handbook  is  intended  as  a  guide  for  those  who  design,  specify,  use, 
and  test  weapon  systems  containing  monitoring  sensors.  The  handbook 
addresses  measures  and  principles  of  measurement,  data  acquisition, 
sensor  calibration  and  testing,  environmental  considerations,  stability, 
durability,  reliability,  and  error  assessment  for  various  types  of  sensors. 

Automatic  Test  Equipment  (ATE) 

1 .  Modular  Automatic  Test  Equipment  (MATE)  Guides 

Although  Air  Force-oriented,  these  guides  describe  procedures  and 
techniques  for  acquiring  automatic  test  equipment.  Of  particular  interest  is 
Guide  5,  which  addresses  the  acquisition  of  test  program  sets. 

2.  MIL-STD-2077,  General  Requirements,  Test  Program  Sets 
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This  standard  covers  the  acquisition  of  test  program  sets  for  use  with  ATE. 
Design  criteria  is  included,  which  addresses  many  detail  requirements  for 
TPSs. 

Human  Engineering 

1.  MIL-STD-1472,  Human  Engineering  Design  Criteria  for  Military  Systems, 
Equipment,  and  Facilities 

This  standard  covers  general  human  engineering  design  criteria  which  can 
be  applied  to  any  weapon  system. 

Technical  Information 

There  are  a  variety  of  standards  which  address  the  preparation  of  technical 
publications.  Most  of  these  documents  are  directed  at  a  specific  military  service.  All 
address  the  delivery  of  paper-type  documentation.  There  is  no  firm  guidance  relating  to 
other,  perhaps  more  innovative  means  for  generating  and  delivering  technical  information. 
In  the  past,  many  technical  publications  have  been  cited  to  have  deficiencies.  These 
deficiencies  can  best  be  described  in  the  DoD  Audit  Report  No.  87-115,  April  3,  1987, 
"Summary  Report  on  the  Defense- Wide  Audit  on  Acquisition  of  Technical  Manuals  and 
Related  Data  From  Contractors." 

Means  should  be  sought  for  generating  and  delivering  this  technical  information 
in  a  less  costly  manner,  without  compromising  its  quality.  There  are  a  number  of  tools 
available,  or  under  development,  which  can  assist  the  designer  of  this  technical 
information  in  authoring  the  text,  when  electronic  delivery  of  technical  information  is 
contemplated.  Some  guidelines  and  standards  for  automatic  generation  of  technical 
information  and  its  delivery  electronically  can  be  obtained  from  the  Human  Resources 
Laboratory  at  Wright-Patterson  Air  Force  Ease.  This  guidance  information  has  been 
developed  under  the  Integrated  Maintenance  Information  System  (IMIS)  Program. 

Innovative  ideas  for  displaying  this  technical  information  are  encouraged,  as 
stipulated  in  Task  303,  MIL-STD- 1388-1.  Care  should  be  taken  to  provide  for  quick 
access  to  the  required  data.  For  electronic  delivery  of  this  data,  formats  may  vary 
substantially  from  paper-based  technical  manuals.  Previously  specified  access  times  and 
information  modification  times  should  influence  the  type  of  generation  and  delivery 
methods.  DoD-lnstruction  4151.9  requires  the  services  to  plan  and  schedule  the 
acquisition  of  technical  manuals  (technical  information)  to  ensure  their  availability  in  final 
form  before,  or  concurrently  with,  delivery  of  the  system  to  the  field.  During  design,  final 
plans  should  be  developed,  along  with  the  support  equipment  which  is  furnished. 
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Maintenance  Aids 

There  Is  a  need  to  present  technical  information  and  troubleshooting  advice  to 
the  technician  on  location  and  readily  available  for  his  use.  The  maintenance  aid, 
sometimes  called  a  job  performance  aid,  presents  information  generated  by  experts  to 
assist  the  less-experienced  technician. 

The  maintenance  aid  is  a  device,  publication,  or  guide  used  on  the  job  to 
facilitate  performance  of  maintenance.  It  delivers: 

o  Historical  information  on  what  fault  was  found  when  similar  symptoms  were 
experienced 

o  Troubleshooting  logic  to  assist  in  finding  the  fault 

o  Procedural  information  which  assists  the  technician  in  finding  and  correcting 
a  failure. 

Normally,  a  maintenance  aid  is  used  in  conjunction  with  a  testing  capability. 
Maintenance  aids  could  be  paper-based  or  employ  electronic  delivery  systems. 

Electronic  delivery  of  this  type  of  information  opens  the  door  to  solving  some  of 
the  problems  associated  with  paper  maintenance  aids.  Two  attributes  of  electronic 
delivery  systems  are: 

o  Information  can  be  available  to  the  technician  in  a  matter  of  seconds  by 
carefully  constructed  menus,  in  lieu  of  the  technician  having  to  page  through 
a  paper  document. 

o  The  collection  of  historical  data  and  the  subsequent  modification  to  the 
software  programs  which  deliver  this  technical  information  can  be  updated 
in  a  matter  of  seconds,  instead  of  a  matter  of  months. 

This  latter  attribute  lends  itself  to  the  introduction  of  expert  systems,  which  often 
employ  artificial  intelligence  technology.  The  expert  system  has  the  capability  of 
combining  various  pieces  of  information  to  lead  the  technician  to  a  logical  decision  on  what 
is  faulty  and  how  it  can  be  repaired. 

Another  important  aspect  of  the  maintenance  aid  is  Its  ability  to  train  technicians 
on  the  job.  Training  programs  must  be  closely  associated  with  the  design  and 
development  of  a  maintenance  aid. 

Over  the  past  20  years,  many  maintenance  aids  have  been  designed, 
developed,  and  tested.  These  tests,  for  the  most  part,  have  proven  successful.  However, 
the  transition  of  these  maintenance  aids  into  the  field  has  not  been  accomplished  to  any 
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great  extent.  One  of  the  reasons  is  that  specifications,  standards,  and  guidance 
information  on  how  to  invoke  this  requirement  are  lacking. 

A  few  important  facts  should  be  remembered  when  applying  maintenance  aids 
and  expert  systems. 

o  The  design  of  the  maintenance  aids  must  be  done  with  the  user  in  mind. 
Once  a  working  model  of  the  equipment  is  available,  there  should  be  a 
dynamic  interchange  of  information  between  the  maintenance  technician 
and  the  design  engineer,  to  ensure  an  effective  and  efficient  man-machine 
interface  is  attained. 

o  User  acceptance  and  adoption  of  maintenance  aids  will  be  facilitated  in 
cases  where  potential  users  are  given  a  trial  period  in  which  to  become 
familiar  with  these  devices  prior  to  their  formal  implementation. 

o  A  system  must  be  devised  to  assure  timely  updating  of  information  to 
correct  errors  and  to  add  newly  acquired  information.  Without  such  a 
system,  the  maintenance  aid  will  quite  rapidly  become  obsolete. 

Manpower  and  Training 

After  personnel  and  training  requirements/allocations  have  been  made,  the 
training  curriculum  needs  to  be  established* concurrently  with  the  system  detail  design. 
This  includes  the  formal  schooling  curriculum,  as  well  as  on-the-job  training.  One  of  the 
alternatives  available,  if  electronic  delivery  of  technical  information  is  employed,  is 
combining  training  aids  with  the  delivered  technical  information.  These  two  types  of 
information  (aiding  and  training)  are  somewhat  similar  in  nature  and,  at  times, 
indistinguishable.  The  training  curriculum  should  be  aimed  at  the  user(s)  and  accessed  in 
a  manner  which  can  be  utilized  by  a  variety  of  users. 

These  training  devices  can  be  freestanding  or  embedded  in  the  prime  system. 
Separate  and  distinct  training  devices  (maintenance  trainers)  may  be  required  to  be 
developed  for  the  formal  schooling. 

Integration  of  Diagnostic  Elements 

There  are  a  variety  of  ways  in  which  diagnostic  elements  can  be  Integrated  to 
produce  a  more  effective  and  efficient  diagnostic  capability.  Expert  system  technology  can 
be  incorporated  in  either  ATE  or  BIT  to  supplement  the  basic  testing  capability.  Fault- 
tolerant  design  and  testability  design  can  be  introduced  into  prime  system  architecture  to 
promote  ease  of  testing,  along  with  graceful  degradation.  Maintenance  aiding  and 
maintenance  training  can  be  combined  to  provide  on-the-job  maintenance  and  training 
information,  utilizing  a  single  portable  device  or  embedded  into  the  prime  system.  Several 
comprehensive  examples  of  this  integration  appear  in  the  Design  Encyclopedia  Guide. 
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CHECKLIST 


Is  the  contractor  utilizing  available  design  guidance? 

□f  Is  the  contractor  attempting  to  integrate  the  various 
diagnostic  elements  to  provide  a  more  effective  and 
efficient  diagnostic  capability? 


ASSESSMENT 


REQUIREMENT  #4 


ANALYSIS  AND  ASSESSMENT  OF  THE  PERFORMANCE  OF  THE  DIAGNOSTIC 


CAPABILITY 

OVERVIEW 


Throughout  the  design  of  the  weapon  system's 
diagnostic  capability,  it  is  essential  to  analyze, 
assess  and  demonstrate  its  performance.  Such 
assessments  are  an  integral  part  of  logistics, 
reliability,  maintainability,  testability,  human 
engineering,  software  and  safety  programs.  The 
ability  to  properly  conduct  these  analyses, 
assessments,  and  demonstrations  is  hampered  by 
the  lack  of  available  techniques  and  tools  to  help, 
and  the  incompatibility  of  the  available  tools  and 
techniques  to  function  together.  Thus  both  the 
program  manager  and  the  designer  must  have 
sufficient  knowledge  to  understand  the  processes 
utilized  and  integrate  these  processes  and  tools  to 
do  the  best  possible  job. 


TESTABILITY/ 

DIAGNOSTICS 


PROGRAMMATIC 


REQUIREMENTS 


DESIGN 


EVALUATION 


MATURATION 


IMPORTANT  CONSIDERATIONS  TO  BE  ADDRESSED 


Roamt 

4.1  Analysis  and  assessment  of  the  diagnostic  capability  should  be 
performed  for  the  entire  diagnostic  capability,  as  well  as  for  each 
diagnostic  element 

4.2  Maintainability  demonstrations  should  be  designed  to  verify  that 
diagnostic  requirements  have  been  met 
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WEAPON 
SYSTEM 
ACQ.  PHASE 


WEAPON 

SYSTEM 

ACTIVITIES 


OPER. 

REQMTS. 


CONCEPT 

EXPLOR. 


DEM/ 

VAL 


PROD/ 

DEPL. 


SYSTEM  PREL.  DETAIL 
SPEC.  DESIGN  DESIGN 


DIAGNOSTIC 

ACTIVITIES 


A  A 


IN-PROCESS  ASSESSMENTS 


DIAGNOSTIC  ACTIVITY 

During  Dem/Val  and  FSD,  it  is  important  to  assess  whether  the 
testabiiity/diagnostic  requirements  are  being  achieved.  This  activity  encompasses  all 
preliminary  and  full-scale  engineering  activities  pertaining  to  both  the  embedded  and 
external  diagnostic  capability. 

PROCEDURE 

In-process  testability/diagnostic  analyses  can  be  conducted  at  most  any  time 
within  Dem/Val  and  FSD.  These  in-process  analyses  are  typically  reviewed  by  the 
government  at  preliminary  design  reviews  and  critical  design  reviews.  These  analyses 
are,  for  the  most  part,  implemented  per  MIL-STD-2165  (T ask  202,  Preliminary  Design,  and 
Task  203,  Detail  Design).  Normally,  these  analyses  will  be  the  responsibility  of  the  design 
or  test  engineer.  However,  it  remains  the  job  of  the  Government  and  Contractor  Program 
Managers  to  understand  the  processes  utilized  and  to  interpret  the  results  of  these 
analyses. 
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GUIDANCE 


Basically,  there  are  two  types  of  in-process  analyses.  The  first  deals  with  the 
inherent  testability  of  the  hardware  design  and  is  independent  of  test  stimuli  and  response 
data.  The  second  type  deals  with  the  effectiveness  of  the  diagnostic  capability  which 
deals  with  measures  that  include  consideration  of  hardware  design,  embedded 
diagnostics,  and  external  diagnostics.  Diagnostic  effectiveness  measures  include,  but  are 
not  limited  to,  fault  coverage,  fault  resolution,  fault  detection  time,  fault  isolation  time,  and 
false  alarm  rate. 

There  are  a  number  of  techniques  and  tools  available,  both  automatic  and 
manual,  which  can  be  used  to  assist  in  these  analyses.  Two  documents  describe  in  detail 
these  techniques  and  the  application  of  these  tools.  The  first  is  the  Testability  Analysis 
Handbook  which  is  described  under  Requirement  #3.2.  The  second  is  the  Diagnostic 
Design  Encyclopedia  (a  companion  document  to  this  Contractor  Program  Managers 
Guide)  which  describes  testability  and  diagnostic  tools  and  their  utilization.  However,  the 
Contractor  Program  Manager  must  understand  the  techniques  and  tools  available  to 
conduct  these  in-process  analyses. 

INHERENT  TESTABILITY 

The  first  analysis  deals  with  inherent  testability.  Inherent  testability  assessment 
is  an  evaluation  of  how  well  a  design  supports  the  testing  process,  whether  built-in  test  or 
off-line  test.  The  evaluation  is  performed  on  the  preliminary  design  and  is  performed 
before  any  test  design  is  performed.  It  is,  therefore,  based  solely  upon  the  hardware 
design  features,  such  as  physical  and  electrical  partitioning,  controllability,  observability, 
and  test  point  placement,  etc.  The  key  to  performing  an  inherent  testability  assessment  is 
the  identification  of  features  which  support  or  inhibit  the  diagnostic  process  early,  at  a  point 
in  time  when  the  design  can  be  changed  relatively  easily.  The  concept  and  the 
implementation  of  an  inherent  testability  assessment  can  have  great  impact  on  overall 
system  supportability. 

In  general,  three  generic  groups  of  inherent  testability  predictive  techniques  are 
available,  each  with  its  unique  advantages  and  disadvantages.  Checklists  are  low  cost, 
manual,  and  somewhat  simplistic.  Logic  models  utilize  the  actual  circuit  topology  but  often 
regard  everything  as  a  block,  with  inputs  and  outputs.,  The  more  detailed  algorithmic 
approaches,  such  as  Sandia  Controllability/Observability  Analysis  Program  (SCOAP), 
require  libraries  of  the  devices  that  most  nearly  simulate  actual  circuit  devices. 

Checklist  approaches  to  inherent  testability  assessment  have  some  very  good 
characteristics,  yet  also  have  some  major  drawbacks.  Checklists  are  manual  approaches 
to  testability  assessment,  yet  are  easily  automated  into  an  interactive  format  for  the 
designer  to  input  answers  to  the  stated  design  criteria,  with  an  automated  grading  being 
done.  However,  quite  extensive  engineering  analysis  is  still  required.  Two  of  these,  the 
RADC  PCB  checklist  and  the  MIL-STD-2165  Appendix  B  checklist,  are  examples.  The 
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RADC  PCB  checklist  is  limited  to  digital  board  applications,  whereas  MIL-STD-2165 
Appendix  B  covers  analog,  digital,  and  hybrid  applications  from  module  to  system  level. 
The  RADC  checklist  has  fixed  items  of  weighting,  whereas  MIL-STD-2165  Appendix  B 
allows  subjective  treating  of  items  and  weighting  values.  Both  items  can  be  utilized  well  in 
selective  applications,  but  the  RADC  checklist  cannot  accommodate  more  complex  digital 
UUTs  and  the  subjectivity  of  MIL-STD-2165  Appendix  B  has  seen  some  criticism  due  to  its 
variable  weighting  scheme. 

Logic  models  have  considerable  success  and  validity  other  than  in  support  of 
the  testability  discipline,  including  logistics,  fault  isolation,  integrated  diagnostics,  and 
maintainability.  The  logic  model  algorithms  are  of  varying  sophistication  and  validity, 
although  the  methodology  for  defining  dependencies  are  similar. 

Logic  models  systems  for  testability  are  applicable  to  analog,  digital,  and  hybrid 
applications.  They  can  be  modeled  at  the  component,  board,  or  module  subsystem  and 
system  level.  One  limitation  of  this  broad  approach  is  that  every  item  is  identified  as  a  box 
with  inputs  and  outputs.  Thus,  box  complexity  might  range  from  an  OR  gate  to  a  complete 
microprocessor.  The  same  variation  applies  to  the  lines  between  boxes.  Critical  signals, 
such  as  a  clock  or  a  tri-state  bus  are  not  more  important  than  any  other  line.  Two  of  the 
more  well-known  models  are  Logic  Modeling  (LOGMOD)  and  System  Testability  Analysis 
Maintenance  Program  (STAMP).  Both  are  mature  in  nature,  but  each  is  tied  to  a  specific 
vendor  at  the  present  time. 

Algorithmic  approaches  are  perhaps  the  most  sophisticated  approach.  SCOAP 
seems  to  usually  perform  well,  but  has  had  some  library  limitations  in  the  important  area  of 
CMOS  primitives.  Some  CAE  workstation  vendors  are  including  modified  versions  of 
SCOAP  for  up-front  testability  analyses.  Daisy  workstations  include  the  Daisy  Testability 
Analyzer  (DTA)  package,  and  G  E/C  ALMA  workstations  include  the  Controllability- 
Observability-Predictability-Testability  Report  (COPTR)  package.  Both  have  improved  on 
the  basic  SCOAP,  via  top-down  modeling  and  large  device  model  libraries  of  the  more 
common  1C  types.  GenRad  also  has  a  package  called  HITAP,  which  is  based  on  the 
Computer-Aided  Measure  for  Logistic  Testability  (CAMELOT)  algorithm. 

Another  major  issue  surrounding  inherent  testability  assessment  is  that  many  of 
the  automated  tools  which  exist  are  proprietary.  This  proprietary  nature  of  the  tools 
creates  implementation  problems  from  both  a  cost  and  a  contractual  point  of  view.  Often, 
the  best  approach  is  to  utilize  a  nonproprietary  automated  tool  for  inherent  testability 
assessment. 

Prior  to  the  FSD  phase,  the  design  or  test  engineer  should  develop  a  total 
strategy  for  conducting  inherent  testability  assessment  on  all  systems,  subsystems,  etc. 
Based  upon  the  availability  and  applicability  of  current  inherent  testability  assessment 
approaches,  it  is  anticipated  that  a  combination  of  tools  and  techniques  will  be  required  to 
form  a  totally  comprehensive  measurement  capability.  In  areas  where  an  automated 
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capability  is  not  available,  use  the  baseline  of  existing  models  to  make  modifications  to 
provide  the  total  capability  required. 

An  evaluation  criteria  for  inherent  testability  assessment  tools  and  techniques 
should  be  developed  based  upon  specific  system  and  subsystem  specific  needs.  The 
following  list  of  evaluation  criteria  is  recommended: 

o  Automation;  degree  of  automation 

o  Proprietary  nature 

o  User  friendliness 

o  Automated  link  to  design  data  base 

o  Acceptability  of  output  to  the  government 

o  Cost  of  use 

o  Availability  (currently  available  or  under  development) 
o  Quality 

o  Sensitivity  to  key  testability  features 
o  Feedback  provided  (does  it  recommend  design  fixes?) 
o  Comprehensiveness  (digital,  analog,  RF,  VHSIC,  mechanical,  etc.) 
o  General  techniques;  principles  used 
o  Link  to  test  effectiveness  prediction  technique 
o  Output  reports 
o  Scoring  methodology 
o  Applicability  to  chip,  board,  subsystem. 

TEST  EFFECTIVENESS 

The  second  type  of  analysis  deals  with  test  effectiveness.  Traditional 
approaches  to  determining  test  effectiveness  call  for  the  generation  of  test  sequences  at 
the  completion  of  the  design  phase  and  a  measure  or  measures  made  of  their 
effectiveness.  The  analysis  need  not  wait  on  the  completion  of  BIT  and/or  off-line  TPS 
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software.  Modeling  is  encouraged,  since  this  approach  can  analyze  test  effectiveness  on 
a  large  number  of  postulated  faults  prior  to  incorporating  the  test  stimulus  in  either  an 
embedded  or  off-line  program.  The  results  of  the  analysis  can  feed  forward,  so  as  to 
influence  BIT  or  TPS  software  design,  and  feed  backward  to  influence  possible  redesign 
of  the  prime  system  to  improve  its  testability.  Test  effectiveness  measures  have 
traditionally  included: 

o  Fault  coverage 

o  Fault  resolution 

o  Fault  detection  time 

o  Fault  isolation  time. 

Computer  programs  are  used  to  input  (via  software)  a  large  number  of  faults  into 
the  software  model  of  the  hardware  item  (UUT).  The  response  of  the  simulated  item  to  the 
test  sequence  is  then  evaluated,  given  the  presence  of  the  simulated  faults.  Fault 
detection,  resolution,  etc.,  are  automatically  ascertained.  Most  modern  Automatic  Test 
Program  Generation  (ATPG)  and  simulation  systems  have  very  efficient  fault  simulation 
capabilities.  The  HITS  system,  for  example,  runs  a  concurrent  fault  simulation  to  greatly 
speed  the  process.  The  usefulness  of  this  approach  in  measuring  test  effectiveness 
depends  on  the  adequacy  of  the  models  (hardware  item  model  and  fault  model)  to 
accurately  reflect  the  real-world  situation.  Modeling  must  be  achieved  at  a  level  of  detail 
that  allows  all  known  and  statistically  significant  failure  modes  to  be  included. 

Although  commonly  accepted,  the  application  of  these  measures  is  In  various 
stages  of  maturity,  based  upon  the  equipment  composition  (i.e.,  digital,  analog,  radio 
frequency  and/or  mechanical).  At  this  time,  the  application  experience  has  been 
concentrated  in  the  area  of  digital  implementations.  However,  even  in  this  area,  significant 
additional  effort  will  be  required  in  order  to  relate  these  measures  to  operational 
performance.  The  degree  of  application  of  test  effectiveness  measurement  techniques  to 
the  remainder  of  the  listed  equipment  types  has  been  quite  limited  to  date.  IDSS,  the 
Navy’s  Integration  Diagnostics  Program,  has  recognized  this  need  and  has  an  active 
diagnostic  tool  development  program  underway.  One  of  these  tools,  the  Weapon  System 
Testability  Analyzer,  is  structured  to  address  test  effectiveness  measurement,  as  well  as 
inherent  testability  assessment. 

Effective  and  realistic  fault  modeling  is  a  key  element  in  the  development  of  the 
simulation  capability  needed  to  support  the  development  of  either  an  ATPG  or  an 
automated  test  effectiveness  measurement  tool.  However,  K  is  anticipated  that  no  single 
fault  model  and/or  simulator  will  be  applicable  to  the  broad  range  of  equipments  to  be 
employed  within  a  complex  system;  therefore,  a  combination  of  models  will  be  required  to 
meet  the  requirement  for  automated  determination  of  fault  detection  and  isolation  levels. 
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CHECKLIST 


O'  Does  the  analysis  of  testability/diagnostic  requirements 
address  all  major  support  disciplines? 

Off-line  ATE 
Embedded  diagnostics 

Manpower  required  to  support  analysis  outputs 
Training  requirements 
Information  requirements. 

□f  Are  all  analyses  complete  and  unambiguous? 

Do  they  meet  specification  requirements? 

□f  Is  the  analysis  integrated  and  cohesive?  Are  any 
requirements  in  conflict? 

Are  the  training,  information,  and  manpower  require¬ 
ments  adequately  scoped  ahd  specified  to  support  the 
technical  complexity  of  the  subject  end  item  in  its 
operational  environment? 

□r  What  design  automation  tools  (e.g.  simulators, 
analyzers)  has  the  contractor  proposed/used  to 
verify  the  predicted  diagnostic  performance? 
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WEAPON 
SYSTEM 
ACQ.  PHASE 

OPER. 

REQMTS. 

CONCEPT 

EXPLOR. 

DEM/ 

VAL 

FSD 

PROD/ 

DEPL. 

WEAPON 

SYSTEM 

ACTIVITIES 

a 

IOT&E 

DIAGNOSTIC 

ACTIVITIES 

A 

Tf  DEMO 

DIAGNOSTIC  ACTIVITY 


Maintainability  Demonstrations  are  performed  In  accordance  with  the 
appropriate  demonstration  method  contained  in  MIL-STD-471A.  Notice  2  of  MIL-STD- 
471 A  (USAF)  contains  requirements  for  demonstration  and  evaluation  of  system 
BIT/external  test/fault  isolation/testability  attributes.  This  method  will  demonstrate  the 
integration  of  the  diagnostic  capability  for  the  system  (e.g.,  integration  of  embedded  test 
software  and  hardware  techniques,  automatic  and  manual  test,  BIT/SIT,  training  levels, 
human  interfaces).  The  Maintainability  Demonstrations  evaluate  the  diagnostic 
performance  of  the  system  with  respect  to  the  diagnostic  performance  criteria  and 
objectives  established  In  accordance  with  MIL-STD-470  (Maintainability  Program)  and 
MIL-STD-2165  (Testability  Program)  and  the  requirements  for  an  "integrated"  diagnostics 
capability  demonstration  contained  in  the  FSD  SOW. 

PROCEDURE 

The  integrated  diagnostics  process  increases  the  scope  of  maintainability 
demonstrations.  It  is  the  Contractor  Program  Manager’s  responsibility  to  ensure  that  this 
increased  scope  is  understood  and  implemented. 

The  scope  of  Maintainability  Demonstrations  includes: 

1 .  Demonstration  of  Testability  Parameters 

-  BIT  Fault  Detection 

-  BIT  Isolation  Time 

-  BIT  Fault  Isolation  Level  (Ambiguity  Group) 
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2.  Demonstration  of  Test  Effectiveness  (ATE)  (MIL-STD-2077) 

-  ATE  Fault  Detection 

-  ATE  Fault  Isolation  Time 

-  ATE  Fault  Isolation  Level  (Ambiguity  Group) 

-  UUT/ATE  Compatibility 

3.  Demonstration  of  Technical  Information 

-  Technical  Information  Access  Time 

-  Technical  Information  Relative  Access  Ease 

-  Technical  Information  Format 

-  Technical  Information  Usability 

4.  Demonstration  of  Training/Skills 

•  Relationship  between  maintenance  procedures  and  skills 

-  Relationship  between  formal  training  and  actual  maintenance  job 
flow. 

5.  Demonstration  of  Vertical  and  Horizontal  Integration 

•  Compatibility  and  Consistency  of  diagnostic  demonstration  results 
between  maintenance  levels  and  among  their  respective  diagnostic 
elements. 

GUIDANCE 


Unfortunately,  the  ability  to  carry  out  a  single  demonstration,  or  even  a 
series  of  demonstrations,  to  prove/evaluate  this  level  of  diagnostic  capability  is 
dependent  upon  having  all  of  the  diagnostic  elements  available  for  the 
maintainability  demonstration.  While  this  should  always  be  the  goal,  it  may  not  be 
feasible  for  all  of  the  above  due  to  development  schedules,  UUT  design  instability, 
data  availability  and  other  overall  program  constraints.  (Note  that  this  is  a  primary 
reason  for  a  Diagnostics  Maturation  Program.) 

Typically,  the  contractor  prepares  a  Maintainability  Demonstration  Plan 
early  in  the  FSD  Phase  and  that  plan  is  subject  to  government  review  and  approval. 
The  Contractor  Program  Manager  should  take  advantage  of  this  opportunity  to 
maximize  the  scope  and  effectiveness  of  the  Maintainability  Demonstrations  to 
include  the  factors  cited  above.  This  can  have  a  significant  cost-savings  impact  on 
the  Diagnostics  Maturation  Program  requirements.  Maintainability  Demonstrations 
represent  the  first  major  opportunity  to  evaluate  the  level  of  diagnostic  capability 
achieved.  Also,  Maintainability  Demonstrations  can  be  conducted  early  enough  to 
implement  corrective  action  cost-effectively.  Demonstrations  are  conducted  while 
the  system  is  still  considered  to  be  in  the  Development  Phase.  After  the 
demonstrations  are  completed,  the  relative  cost  of  identifying  deficiencies  and 
implementing  corrective  action  is  significantly  increased.  A  significant  milestone  of 
'Government  Acceptance'  occurs  upon  satisfactory  completion  of  Maintainability 
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Demonstrations.  After  this  milestone,  costs  for  identification  and  resolution  of 
diagnostic  deficiencies  may  be  subject  to  contract  interpretation  and/or  negotiation. 
The  total  strategy  for  the  test  and  evaluation  of  the  diagnostic  capability  is  placed  on 
the  TEMP,  and  detailed  in  the  Integrated  Test  Plan. 

Based  upon  the  selected  scope  of  the  Maintainability  Demonstration, 
procedures  from  MIL-STD-471  are  utilized  and  adapted  for  the  scope.  These 
procedures  are  documented  in  the  Maintainability  Demonstration  Plan.  The  results 
of  the  Maintainability  Demonstration  are  documented  in  a  technical  report  - 
Maintainability  Demonstration  Results. 

Concurrent  Demonstrations 

The  overall  diagnostic  capability  is  the  sum  of  a  variety  of  diagnostic 
elements.  Therefore,  a  requirement  should  be  established  for  early  demonstration 
of  the  entire  diagnostic  capability  produced  by  the  integration  of  all  of  these 
diagnostic  elements.  This  is  referred  to  as  concurrent  demonstrations,  where  the 
timing  of  various  diagnostic  element  demonstrations  are  planned  and  scheduled  for 
concurrency  so  that  the  integrated  capability  can  be  assessed. 

The  Contractor  Program  Manager  must  evaluate  schedules  to  determine 
the  level  of  concurrency  feasible.  Critical  and/or  risk  areas  must  be  identified  and 
evaluated. 


Each  element  of  the  diagnostic  capability  must  be  demonstrated,  as  well 
as  the  result  of  the  combining  or  integration  of  the  elements.  For  example,  a 
demonstration  of  subsystem  BIT  may  prove  fault  detection  and  isolation  levels.  A 
demonstration  of  ATE  may  prove  external  fault  detection  and  isolation  levels.  A 
concurrent  demonstration  of  these  two  diagnostic  elements  will  prove  the  ability  of 
the  ATE  to  use  BIT  circuitry,  to  use  BIT  results,  and  the  consistency  of  test  results 
between  BIT  and  ATE.  By  concurrent  demonstration,  the  whole  is  greater  than  the 
sum  of  the  parts.  A  significant  set  of  factors  related  to  the  result  of  the  integration  of 
the  diagnostic  elements  must  be  evaluated  early. 
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CHECKLIST 


O'  Does  the  demonstration  plan  provide  a  100%  fault 
coverage  capability  across  all  levels  of  maintenance? 

o  Organizational  Level 
o  Intermediate  Level 
o  Depot  Level 

o'  Are  the  failure  modes  to  be  demonstrated  and  criteria 
to  be  utilized  adequately  specified  for  each 
maintenance  level? 

of  Does  conducting  the  actual  demonstration  require  a 
level  of  manpower,  training,  and/or  technical  informa¬ 
tion  above  and  beyond  that  which  will  be  provisioned? 
If  so,  why? 

rf  Is  the  demonstation  structured  to  provide  an 
evaluation  of  the  diagnostic  capabilities  as 
an  integrated  system? 

Do  the  subject  plans  demonstrate  an  integrated, 
cohesive  maintenance  flow  in  terms  of  demonstrating 
how  a  fault  would  be  detected?  For  example,  at  the 
Organizational  Level  and  the  subject  repair  effected  at 
the  Depot  Level?  Is  a  systems  approach  to  the 
maintenance  process  taken  in  the  overall  approach  to 
demonstration  planning? 
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CONDUCTING  DESIGN  REVIEWS 
OVERVIEW 


During  the  acquisition  of  a  weapon  system  there 
are  at  least  eight  formal  technical  reviews  and 
audits,  which  may  be  conducted  by  the  contractor 
for  the  Government  Program  Manager.  As  in  the 
diagnostic  design  process,  there  is  a  tendency  to 
conduct  separate  reviews  and  audits  based  upon 
the  function  being  addressed.  This  particularly 
refers  to  logistics,  reliability,  maintainability, 
testability,  human  engineering,  and  safety. 
Integration  of  these  reviews  and  audits  to  address 
diagnostic  issues  is  a  must.  MIL-STD-1521  is  the 
prime  document  which  defines  the  issues  to  be 
addressed  at  each  of  these  formal  reviews.  At 
present,  these  checklists  are  inadequate  in  terms 
of  both  testability  and  diagnostics  and,  thus,  these 
reviews  and  audits  may  not  serve  their  purpose. 
Additional  guidance  must  be  given  to  both  the 
government  and  the  contractor  in  order  to  alleviate 
this  problem. 

Informal  reviews  are  often  required.  Guidance  for 
these  informal  reviews  can  be  drawn  from  formal 
review  guidance. 


TESTABILITY/ 

DIAGNOSTICS 


PROGRAMMATIC 


REQUIREMENTS 


DESIGN 


ASSESSMENT 


EVALUATION 


MATURATION 


IMPORTANT  CONSIDERATIONS  TO  BE  ADDRESSED 


Reqmt. 

5.1  Technical  reviews  and  audits  must  address  all  facets  which  affect  the 
performance  of  the  diagnostic  capability. 

Conduct  the  review  and  audit  of  testability  and  diagnostic  functions  as 
an  Integral  part  of  system  engineering  and  maintainability  reviews. 
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WEAPON 

SYSTEM 

ACQ.  PHASE 

OPER. 

REQMTS. 

CONCEPT 

EXPLOR. 

DEM/ 

VAL 

FSD 

PROD/ 

DEPL 

WEAPON 

SYSTEM 

ACTIVITIES 

A AAA A 

SCP  DCP  PREL.  DETAIL  I0T&E 

DESIGN  DESIGN 

DIAGNOSTIC 

ACTIVITIES 

A  AAAAAA  A 

SRR  SDR  PDR  CDR  TRR  PRR  FCA  PCA 

DIAGNOSTIC  ACTIVITY 

Technical  reviews  and  audits  are  an  important  factor  in  assuring  that  the 
government  is  furnished  with  a  weapon  system  which  meets  its  requirements. 

PROCEDURE 


MIL-STD-1521  lists  10  formal  technical  reviews  and  audits.  Of  these  10,  eight 
are  considered  critical  in  the  achievement  of  a  satisfactory  diagnostic  capability.  The 
following  guidance  supplements  and  expands  the  guidance  contained  in  MIL-STD-1521, 
Technical  Reviews  and  Audits  for  Systems,  Equipments,  and  Computer  Software. 

Although  design  reviews  are  recognized  as  being  important  to  verify  design 
before  production,  the  lack  of  depth  in  these  reviews  is  alarming.  The  cause  of  these 
inadequate  reviews  must  be  shared  by  both  the  contractors  and  the  government. 
Contractually,  the  government  rarely  requires  the  contractor  to  do  a  comprehensive 
technical  review,  and  the  contractor  does  not  do  so  unless  required,  even  though  it  may 
be  cost  effective  from  his  point  of  view.  Even  when  the  right  words  are  used,  the  end 
results  depend  largely  on  corporate  policy  to  allocate  sufficient  resources  to  perform  a 
detailed  analysis  of  the  design  and  associated  processes. 
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GUIDANCE 


Guidance  relating  to  these  various  reviews  is  contained  in  the  appendices  to 
MIL-STD-1521 .  Because  these  appendices  do  not  address  all  aspects  of  testability  and 
diagnostics,  some  supplemental  information  is  included  in  the  following  paragraphs. 

System  Requirements  Review  (SRR) 

The  objective  of  this  review  is  to  ascertain  the  adequacy  of  the  contractor’s 
efforts  in  defining  system  requirements.  It  will  be  conducted  when  a  significant  portion  of 
the  system  functional  requirements  has  been  established. 

The  diagnostic  capability  review  portion  of  the  SRR  will  analyze  the  system 
items  that  are  related  to  diagnostics.  The  following  items  will  be  reviewed,  as  appropriate: 

o  Mission  and  Requirements  Analysis 
o  Functional  Flow  Analysis 
o  Preliminary  Requirements  Allocation 
o  System/Cost  Effectiveness  Analysis 
o  Trade  Studies 
o  Synthesis 

o  Logistic  Support  Analysis 
o  Specialty  Discipline  Studies 
o  Generation  of  Specifications 
o  Program  Risk  Analysis 
o  Integrated  Test  Planning 
o  Technical  Performance  Measurement  Planning 
o  Engineering  Integration 
o  System  Safety 
o  Human  Factors  Analysis 
o  Life  Cycle  Cost  Analysis 
o  Manpower  Requirements/Personnel  Analysis 
o  Milestone  Schedules. 

The  diagnostic  capability  review  should  address  the  impact  of  the  results  of  the 
Hems  listed  above  on  the  diagnostic  pieces  listed  below. 

o  Designed-In  Reliability,  Prognostics,  and  Testability 
o  Self-Test,  BuiK-ln  Test,  System  Integrated  Test 
o  Support  Equipment,  Maintenance  Aids 
o  Technical  Data 
o  Personnel  Skill  Requirements 
o  Training  and  Training  Devices. 
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System  Design  Review  (SDR) 

This  review  shall  be  conducted  to  evaluate  the  optimization,  correlation, 
completeness,  and  risks  associated  with  the  allocated  technical  requirements.  Also 
included  is  a  summary  review  of  the  system  engineering  process  which  produced  the 
allocated  technical  requirements  and  of  the  engineering  planning  for  the  next  phase  of 
effort.  Basic  manufacturing  considerations  will  be  reviewed,  and  planning  for  production 
engineering  in  subsequent  phases  will  be  addressed.  This  review  will  be  conducted  when 
the  system  definition  effort  has  proceeded  to  the  point  where  system  characteristics  are 
defined  and  the  Configuration  Items  (Cl)  are  identified. 

Specific  diagnostic  considerations  relate  to: 

o  Optimizing  the  diagnostic  capability  (changes  after  Dem/Val  usually  are  more 
costly  and  time  consuming) 

o  Preparation  of  a  Maturation  Pl?.n 

o  Preparation  of  a  System  Specification  which  provides  a  capability  for 
addressing  necessary  FD/FI  requirements  at  each  level  of  maintenance 

o  Allocation  of  diagnostic  requirements  for  each  diagnostic  element 

o  Review  of  the  software  requirements  specification  to  assure  that  embedded 
diagnostic  software  considerations  are  included. 

Preliminary  Design  Review  (PDR) 

This  review  shall  be  conducted  for  each  Configuration  Item  or  aggregate  of 
Configuration  Items  to:  (1)  evaluate  the  progress,  technical  adequacy,  and  risk  resolution 
(on  a  technical,  cost,  and  schedule  basis)  of  the  selected  design  approach;  (2)  determine 
its  compatibility  with  performance  and  engineering  specialty  requirements  of  the  Hardware 
Configuration  Item  (HWCI)  development  specification;  (3)  evaluate  the  degree  of  definition 
and  assess  the  technical  risk  associated  with  the  selected  manufacturing 
methods/processes;  and,  (4)  establish  the  existence  and  compatibility  of  the  physical  and 
functional  interfaces  among  the  Configuration  Item  and  other  items  of  equipment,  facilities, 
computer  software,  and  personnel.  For  Computer  System  Configuration  Items  (CSCIs), 
this  review  will  focus  on:  (1)  the  evaluation  of  the  progress,  consistency,  and  technical 
adequacy  of  the  selected  top-level  design  and  test  approach;  (2)  compatibility  between 
software  requirements  and  preliminary  design;  and,  (3)  on  the  preliminary  version  of  the 
operation  and  support  documents. 

In  addition,  the  following  items  in  the  diagnostic  area  should  be  presented  at  the 
appropriate  depth  for  review. 
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o  Preliminary  Failure  Modes  and  Effects  Analysis 

o  Design  data  analyses  for  BIT/SIT  integrated  diagnostics,  including 
requirements  and  preliminary  design  verification  results 

o  Maintenance  concept  for  the  portion  of  the  system  being  reviewed  and  its 
traceability  to  the  system  maintenance  concept 

o  Operational  maintenance  functions 

o  Results  of  the  analysis  of  the  inherent  (intrinsic)  testability  of  the  preliminary 
design 

o  Allocation  of  qualitative  and  quantitative  requirements 
o  Criteria  for  external  diagnostic  elements 
o  Trade-off  studies 

o  Cost/System  Effectiveness  Modeling  and  Life  Cycle  Cost  Analysis 

o  Preliminary  Logistic  Support  Analysis,  including  task  analysis  and  related 
personnel  and  support  equipment  information 

o  Impact  of  diagnostics  on  maintenance  man-hours  required 

o  Evaluation  of  alternatives 

o  Test  and  evaluation  plans. 

Critical  Design  Review  (CDR) 

This  review  shall  be  conducted  for  each  Configuration  Item  when  detail  design  is 
essentially  complete.  The  purpose  of  this  review  will  be  to:  (1 )  determine  that  the  detail 
design  of  the  Configuration  Item  under  review  satisfies  the  performance  and  engineering 
specialty  requirements  of  the  HWCI  development  specifications;  (2)  establish  the  detail 
design  compatibility  among  the  configuration  and  other  items  of  equipment,  facilities, 
computer  software  and  personnel;  (3)  assess  Configuration  Item  risk  areas  (on  a 
technical,  cost,  and  schedule  basis);  (4)  assess  the  results  of  the  producibility  analyses 
conducted  on  system  hardware;  and,  (5)  review  the  preliminary  hardware  product 
specifications.  For  CSCIs,  this  review  will  focus  on  the  determination  of  the  acceptability 
of  the  detailed  design,  performance,  and  test  characteristics  of  the  design  solution,  and  on 
the  adequacy  of  the  operation  and  support  documents.  The  CDR  shall  be  conducted  on 
each  Configuration  Item  prior  to  fabrication/production/coding  release  to  ensure  that  the 
detail  design  solutions,  as  reflected  in  the  Draft  Hardware  Product  Specification,  Software 
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Detail  Design  Document  (SDDD),  Data  Base  Design  Document(s)  (DBDD),  Interface 
Design  Document(s)  (IDD),  and  engineering  drawings,  satisfy  requirements  established  by 
the  Hardware  Development  Specification  and  Software  Top-Level  Design  Document 
(STLDD).  The  CDR  shall  be  held  after  the  Computer  Software  Operator's  Manual 
(CSOM),  Software  User’s  Manual  (SUM),  Computer  System  Diagnostic  Manual  (CSDM), 
Software  Programmer’s  Manual  (SPM),  and  Firmware  Support  Manual  (FSM)  have  been 
updated  or  newly  released. 

It  is  desired  at  each  CDR  to  provide  as  much  assurance  as  practicable  that 
mission-critical  FD/FI  thresholds  are  realized  and  that  all  diagnostic  requirements  are 
satisfied:  e.  g.,  that  100%  diagnostic  capability  will  exist  for  each  Cl  in  the  system.  While 
it  probably  will  not  be  practicable  to  certify  that  this  will  exist,  the  following  data  should  be 
presented  as  an  extension  of  the  information  presented  at  the  PDR. 

o  Detailed  fault  detection/fauit  isolation  analyses  that  identify  the  extent  to 
which  BIT/SIT  detect  and  isolate  faults  and  which  identify  those  failures  that 
will  require  support  equipment  and/or  manual  methods  to  detect  and/or 
isolate. 

o  Diagnostic  allocations  in  Part  II  Cl  specifications  to  the  LRU  and  SRU  level. 
Traceability  of  these  requirements  to  the  Part  I  Cl  System  Specification 
should  be  demonstrated.  Note:  Flexibility  to  reallocate  diagnostic 
allocations  until  product  baseline  is  established  at  PCA  should  be  provided 
within  the  envelope  of  system  requirements. 

o  Definition  of  the  maintenance  plan/concept  for  the  Cl,  together  with 
supporting  LSA  documentation,  including  support  requirement  and  level-of- 
repair  analysis  results.  Logistic  simulation  results  should  be  presented  to 
substantiate  the  plan/concept. 

o  Presentation  of  testability  analysis/assessment  results  for  the  Cl  design  to 
substantiate  the  fault  detection/  fault  isolation  analysis. 

o  Early  program  Failure  Identification,  Prevention,  and  Detection  (FI PAD) 
analyses  applicable  to  the  Cl  should  be  presented  to  assist  in  verifying 
diagnostic  capability. 

o  Review  detailed  Maintainability  Demonstration  Plan  for  inclusion  of 
diagnostic  capability  test  requirements 

o  Appropriate  updates  to  the  items  reviewed  during  the  PDR. 
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Test  Readiness  Review  (TRR) 

This  review  is  conducted  for  each  CSCI  to  determine  whether  the  software  test 
procedures  are  complete  and  to  assure  that  the  contractor  is  prepared  for  formal  CSCI 
testing.  Software  test  procedures  are  evaluated  for  compliance  with  software  test  plans 
and  descriptions  and  for  adequacy  in  accomplishing  test  requirements.  At  TRR  the 
contracting  agency  also  reviews  the  results  of  informal  software  testing  and  any  updates  to 
the  operation  and  support  documents.  A  successful  TRR  is  predicated  on  the  contracting 
agency’s  determination  that  the  software  test  procedures  and  informal  test  results  form  a 
satisfactory  basis  for  proceeding  into  formal  CSCI  testing. 

The  diagnostic  segment  of  the  system/CI  TRR(s)  shall  be  a  formal  review  of  the 
contractor’s  readiness  to  begin  formal  diagnostics-related  CSCI  testing.  It  is  conducted 
after  the  software  test  procedures  are  available  for  diagnostics-related  CSCI,  such  as  Cl 
BIT,  System  BIT,  SIT,  etc.,  and  after  computer  system  component  (CSC)  integration 
testing  is  complete. 

The  items  to  be  reviewed  include: 

1.  Requirement  Changes  -- 

Any  changes  to  BIT,  SIT,  or  testability  requirements  contained  in  the 
system/CI  Software  Requirement  Specification  or  Interface  Requirements 
Specification  that  have  not  been  approved  and  which  impact  CSCI  testing. 

2.  Design  Changes  - 

Any  changes  made  to  the  BIT,  SIT,  or  testability  design  parameters 
contained  in  the  Software  Top-Level  Design  Document  (STLDD),  Software 
Detail  Design  Document  (SDDD),  Interface  Design  Document(s)  (IDD)  since 
the  PDR  and  CDR  which  impact  CSCI  testing. 

3.  Software  Test  Plans  and  Descriptions  — 

Any  changes  to  the  embedded  diagnostic  element  portion  of  the  approved 
Software  Test  Plans  (STP)  and  Software  Test  Descriptions  (STD). 

4.  Software  Test  Procedures  -- 

Test  procedures  to  be  used  in  conducting  BIT  and/or  SIT  test  effectiveness 
validation  as  part  of  the  CSCI  testing,  including  retest  procedures  for  test 
anomalies  and  corrections. 

5.  Integration  Test  Cases,  Procedures,  and  Results  - 
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Any  embedded  diagnostic  element  CSC  (e.  g.,  BIT  components,  SIT 
components)  integration  test  cases,  and  procedures  used  in  conducting 
informal  diagnostic  element  CSC  integration  tests  and  the  test  results. 


6.  Software  Test  Resources  - 

Status  of  any  software  test  resources  that  are  required  specifically  for 
embedded  diagnostic  element  CSC!  testing.  Such  resources  may  include 
diagnostic  test  personnel  and  supporting  test  software  and  materials, 
including  software  test  tool  qualification  and  review  of  the  traceability 
between  requirements  and  their  associated  tests. 

7.  Test  Limitation  -- 

Identification  of  all  software  test  limitations  associated  with  embedded 
diagnostic  element  CSCI/CSC  testing. 

8.  Software  Problems  - 

Summary  of  embedded  diagnostic  element  software  problem  status, 
including  all  known  discrepancies  of  the  CSCI  and  test  support  software. 

9.  Schedules  - 

Schedules  for  the  remaining  embedded  diagnostic  element  software 
milestones. 

Production  Readiness  Review  (PRR) 

This  review  is  intended  to  determine  the  status  of  completion  of  the  specific 
actions  which  must  be  satisfactorily  accomplished  prior  to  executing  a  production  go- 
ahead  decision.  The  review  is  accomplished  in  an  incremental  fashion  during  the  Full- 
Scale  Development  Phase-usually  two  initial  reviews  and  one  final  review,  to  assess  the 
risk  in  exercising  the  production  go-ahead  decision.  In  its  earlier  stages,  the  PRR 
concerns  itself  with  gross-level  manufacturing  concerns,  such  as  the  need  for  identifying 
high-risk/low-yieid  manufacturing  processes  or  materials  or  the  requirement  for 
manufacturing  development  effort  to  satisfy  design  requirements.  The  reviews  become 
more  refined  as  the  design  matures,  dealing  with  such  concerns  as  production  planning, 
facilities  allocation,  incorporation  of  producibility-oriented  changes,  Identification  and 
fabrication  of  tools/test  equipment,  long-lead  item  acquisition,  etc.  Timing  of  the 
incremental  PRRs  is  a  function  of  program  posture  and  is  not  specifically  locked  into  other 
reviews.  The  diagnostic  consideration  concerns  the  use  of  any  of  the  external  diagnostic 
elements  (e.g.,  ATE)  in  the  production  testing  environment. 
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Functional  Configuration  Audit  (FCA) 

This  is  a  formal  audit  to  validate  that  the  development  of  a  Configuration  Item 
has  been  completed  satisfactorily  and  that  the  Configuration  Item  has  achieved  the 
performance  and  functional  characteristics  specified  in  the  functional  or  allocated 
configuration  identification.  In  addition,  the  completed  operation  and  support  documents 
shall  be  reviewed. 

The  FCA  is  normally  conducted  on  a  prototype  or  preproduction  item.  The  FCA 
validates  that  the  item  meets  its  specified  performance  requirements  and  is  ready  for 
production  and  acceptance  into  Air  Force  inventory.  It  is  imperative  that  the  diagnostic 
capability  be  validated  against  its  specified  performance  requirements,  so  that  any 
diagnostic  capability  deficiencies  can  be  identified  and  corrected  before  the  item  proceeds 
into  production  and  is  then  deployed. 

Physical  Configuration  Audit  (PCA) 

This  is  a  technical  examination  of  a  designated  Configuration  Item  to  verify  that 
the  Configuration  Item  "as  built"  conforms  to  the  technical  documentation  which  defines 
the  Configuration  Item. 

After  successful  completion  of  the  audit,  all  subsequent  changes  to  the 
diagnostic  elements  are  processed  by  an  engineering  change  action.  The  PCA  also 
determines  that  the  diagnostic  element  acceptance  testing  prescribed  by  the 
documentation  is  adequate  for  acceptance  of  the  production  units  by  quality  assurance 
activities.  The  procedures  for  conducting  a  PCA  are  contained  in  MIL-STD-1521, 
Appendix  H.  Sample  PCA  Certification  Attachment  Checklists  are  contained  in  MIL-STD- 
1521,  Appendix  I. 
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CHECKLIST 


Is  emphasis  being  placed  on  technical  inter¬ 
change  meetings  between  contractor  and  customer 
rather  than  large-scale  reviews? 

□r  Are  qualified  diagnostic  technical  experts,  who 
can  challenge  the  design  and  assess  risks, 
included  in  these  reviews? 

O'  Are  the  diagnostic  reviews  held  as  an  integral 
part  of  the  prime  system  review,  but  in 
a  timely  manner  that  allows  change  (if  necessary) 
in  the  diagnostic  equipment  or  process? 


EVALUATION 


REQUIREMENT  #6 


CONDUCTING  TEST  AND  EVALUATION 
OVERVIEW 


During  the  development  of  a  weapon  system,  a 
number  of  tests  and  evaluations  are  conducted  by 
subcontractors,  the  prime  contractor,  and  the 
government.  Many  of  these  tests  address  the 
performance  of  the  diagnostic  capability.  It  is  not 
uncommon  that  these  tests  are  conducted 
separately  and,  thus,  do  not  address  the  entire 
diagnostic  capability.  Oftentimes  the  entire 
diagnostic  capability  is  not  delivered  in  time  to  test 
and  evaluate  the  diagnostic  capability  as  a  whole. 
During  the  major  tests  and  evaluations  (e.g., 
DT&E,  OT&E)  as  much  as  possible  of  the  entire 
diagnostic  capability  should  be  included. 
Integrated  demonstration,  test,  and  evaluation  is 
required. 


Coordination  of  all  test  and  evaluations, 
including  demonstrations,  can  be  accomplished 
through  the  preparation  of  an  Integrated  Test  Plan. 


TESTABILITY/ 

DIAGNOSTICS 


PROGRAMMATIC 


REQUIREMENTS 


DESIGN 


ASSESSMENT 


REVIEWS 


MATURATION 


IMPORTANT  CONSIDERATIONS  BE  ADDRESSED 


Reqmt 

6.1  Prepare  an  Integrated  Test  Plan,  which  Includes  the  requirements  for  a 
Test  and  Evaluation  Master  Plan. 

6.2  Assure  that  formal  test  and  evaluations  address  the  entire  diagnostic 
capability. 
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WEAPON 
SYSTEM 
ACQ.  PHASE 

OPER. 

REQMTS. 

CONCEPT 

EXPLOR. 

DEM/ 

VAL 

FSD 

PROD/ 

DEPL. 

WEAPON 

SYSTEM 

ACTIVITIES 

£ £ £ £ 

V - - — - - ^  CDR 

CONTRACT 

AWARD 

DIAGNOSTIC 

ACTIVITIES 

£  £  £ 

ttup  temp  TEMP 

1  "Mr  UPDATE  UPDATE 

DIAGNOSTIC  ACTIVITY 


The  requirements  for  diagnostics  test  and  evaluation  are  identified,  scheduled 
and  integrated  into  the  Test  and  Evaluation  Master  Plan  (TEMP)  by  the  contractor. 

PROCEDURE 


The  Contractor  Program  Manager  should  ensure  that  adequate  requirements 
are  in  place  for  diagnostics  test  and  evaluation.  The  TEMP  is  a  living  document  •  its 
preparation  goes  through  many  iterations  as  the  program  proceeds  through  Concept 
Exploration  Phase  studies,  Demonstration  and  Validation,  Full-Scale  Development, 
Production  and  Deployment.  With  each  iteration,  plans  for  diagnostic  T&E  should  become 
firmer,  better  defined,  and  with  target  milestone  dates  attached. 

Because  test  and  evaluation  is  a  major  cost  and  schedule  driver,  adequate 
planning  is  essential  long  before  its  start.  Test  planning  between  subcontractors,  the 
prime  contractor,  and  the  government  should  start  with  program  initiation.  To  ensure  a 
successful  integrated  test  program,  close  coordination  is  required  between  the 
government,  the  prime  contractor,  and  all  subcontractors. 

GUIDANCE 


DoD  Directive  5000.3  requires  the  preparation  of  a  TEMP.  The  TEMP  is  a 
broad  plan  relating  test  objectives  to  required  system  characteristics  and  critical  issues, 
and  is  a  top-level  document  used  at  major  milestone  reviews  to  assess  the  adequacy  of 
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planned  test  and  evaluation.  The  TEMP  normally  covers  only  government-required  tests, 
and  does  not  provide  a  sufficient  level  of  detail  to  identify  contractor  and  subcontractor 
te«4s.  In  an  attempt  to  control  the  test  program  at  the  contractor  and  subcontractor  level, 
ce  tracts  may  contain  requirements  for  the  submittal  of  individual  test  plans  for 
government  approval.  If  an  Integrated  Test  Plan  is  not  required,  these  individual  test  plans 
may  not  be  reviewed  for  duplicate  or  missing  test  activities,  resulting  in  an  inefficient  and 
costly  test  programs. 

The  prime  contractor  should  be  responsible  for  the  preparation  and  updating  of 
an  Integrated  Test  Plan  (ITP).  To  develop  an  efficient  and  well  coordinated  Integrated 
Test  Plan,  the  prime  contractor  and  all  subcontractors  should  jointly  participate  in  its 
preparation.  The  ITP  should  include  all  developmental  tests  to  be  performed  by  the  prime 
contractor  and  all  subcontractors  at  both  the  system  and  subsystem  levels.  The  ITP 
should  be  a  detailed  working-level  document  which  will  aid  in  identifying  risk,  duplication  or 
missing  test  activities,  and  provide  for  the  most  efficient  use  of  test  facilities  and  test 
resources.  In  developing  the  ITP,  the  purpose  and  time  phasing  of  each  individual  test 
should  be  carefully  examined.  Unnecessary  tests  should  be  eliminated  and  test 
schedules  should  be  adjusted  to  provide  sufficient  time  for  retest,  should  failures  occur. 
The  proper  sequencing  of  tests  is  necessary  to  ensure  completion  of  required  lower-level 
subcontractor  tests  prior  to  the  start  of  prime  contractor  tests.  Detailed  requirements  for 
diagnostics  T&E  should  be  included  in  the  integrated  Test  Plan.  These  requirements 
should  be  phased  T&E  of  all  diagnostic  elements  and  the  integration  of  the  elements.  The 
ITP  should  also  include  close  coordination  between  all  T&E  activities. 

During  Development  Test  and  Evaluation  (DT&E)  the  contractor  and  the 
government  normally  conduct  separate,  dedicated  tests.  In  many  instances  these 
separate  test  periods  result  in  redundant  testing,  testing  which  is  not  user-oriented,  lack  of 
continuity  in  the  contractor’s  development  program,  and  a  lack  of  cooperation  between 
contractor  and  government  personnel.  In  order  to  increase  the  efficiency  of  DT&E  effort, 
where  possible,  should  be  made  to  combine  tests.  This  will  help  eliminate  redundant 
testing,  reduce  the  length  of  DT&E  phases,  provide  more  user-oriented  test  results,  and 
result  in  a  more  mature  system  for  Operational  Test  and  Evaluation  (OT&E). 

Test  schedules  should  be  properly  phased  and  based  on  development 
engineering  considerations.  The  purpose  or  objective  of  each  test  should  be  considered 
as  well  as  the  interrelation  of  various  tests  with  each  other.  As  test  programs  progress, 
many  tests  will  disclose  a  need  for  redesign  and  retest,  such  as  redesign  of  System 
Integrated  Test  (SIT)  or  Built-In  Test  (BIT)  hardware  or  software.  In  some  instances  only 
a  minor  correction  and  verification  test  will  be  required.  In  other  cases  the  corrective 
actions  may  be  extensive  and  require  significant  retest.  If  test  schedules  have  not  allowed 
sufficient  time  for  redesign  and  retest,  changes  and  retesting  may  be  delayed  until 
production  equipment  is  available.  If  the  changes  prove  incorrect  and  additional  redesign 
is  required,  production  units  may  have  to  be  retrofitted  and  a  large  number  of  Engineering 
Change  Proposals  (ECPs)  may  be  required  during  the  early  phases  of  the  production 
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program.  Also,  due  to  the  sequential  nature  of  some  tests,  the  performance  of  certain 
tests  may  be  delayed  until  production,  possibly  resulting  in  additional  ECPs. 

Since  the  start  of  certain  tests  may  be  dependent  upon  the  completion  of  others, 
critical  tests  should  be  identified  and  provisions  made  for  schedule  slippage  due  to  needed 
redesign  and  retest.  In  certain  cases  critical  test  schedules  can  be  accelerated  by 
providing  more  test  assets  or  additional  test  facilities.  This  strategy  can  provide  significant 
leverage  to  reduce  the  overall  development  test  schedule.  Milestone  reviews  can  then  be 
planned  on  the  basis  of  realistic  test  schedules.  More  engineering-oriented  test  results 
showing  design  strengths  and  weaknesses  should  be  presented  at  design  reviews.  The 
review  should  discuss  design  weaknesses  and  how  they  have  been  or  will  be  corrected. 
The  overall  success  of  a  carefully  integrated  test  program  will  result  in  a  minimum  of 
resources  applied  to  testing  and  the  elimination  of  a  costly  ECP  or  retrofit  program  during 
production. 

In  accordance  with  DoD  Directive  5000.3,  test  and  evaluation  must  begin  as 
early  as  possible  in  the  system  acquisition  process  to  reduce  acquisition  risks  and  to 
estimate  the  capability  of  the  system  under  development  to  meet  all  technical  and 
operational  requirements.  Critical  test  and  evaluation  issues  (to  include  all  effectiveness 
and  suitability  issues),  objectives,  methodologies  and  evaluation  criteria  are  defined  during 
the  initial  establishment  of  an  acquisition  program.  These  criteria  serve  to  define  the 
testing  required  for  each  phase  of  the  acquisition  process  and  provide  the  structure  to 
guide  the  testing  program.  Diagnostics  should  be  established  as  a  critical  suitability  and 
logistic  supportability  issue  in  order  to  provide  the  proper  degree  of  focus  on  diagnostics  T 
&  E.  For  example,  if  two-level  maintenance  is  a  system  requirement,  then  diagnostics 
capability  is  very  critical  to  achieving  that  requirement. 

Testing  must  be  planned  and  conducted  to  provide  quantitative  data  and  to 
minimize  the  need  for  subjective  interpretation  of  system  performance  and  suitability 
factors.  This  requires  early  planning  to  determine  the  number  of  test  articles  needed  as 
well  as  other  support  resources.  The  developer  and  the  test  and  evaluation  agencies 
should  share  information  and  data  during  the  acquisition  process  to  establish  a  data  base 
allowing  progressive  evaluation  of  the  system. 

The  use  of  properly  validated  analysis,  models  and  simulation  is  strongly 
encouraged,  especially  during  development  phases  to  assess  the  areas  which  cannot  be 
observed  through  testing.  Use  of  these  methods  can  provide  early  projections  and  can 
reduce  testing  costs  by  supplementing  actual  test  data. 

Developmental  Test  and  Evaluation  (DT&E)  is  the  T&E  conducted  throughout 
various  phases  of  the  acquisition  process.  This  will  ensure  the  acquisition  and  fielding  of 
an  effective  and  supportable  system  by  assisting  in  the  engineering  design  and 
development  and  verifying  attainment  of  technical  performance  specifications,  objectives 
and  supportability. 
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Developmental  Test  and  Evaluation  also  includes  T&E  of  components, 
subsystems,  preplanned  product  improvement  (P^l)  changes,  hardware-software 
integration  and  related  software,  as  well  as  qualification  and  production  acceptance 
testing.  Test  and  evaluation  of  compatibility  and  interoperability  with  existing  or  planned 
equipment  or  systems  is  emphasized.  This  T&E  encompasses  the  use  of  models, 
simulations  and  testbeds,  as  well  as  prototypes  of  Full-Scale  Development  models  of  the 
system.  The  diagnostic  capability  associated  with  component,  assembly  and  subsystem 
DT&E  should  be  included  in  these  T&E  activities. 

Qualification  Testing  is  that  part  of  DT&E  which  verifies  the  design  and  the 
manufacturing  process  and  provides  a  baseline  for  subsequent  acceptance  tests.  This 
accomplishes  two  separate  functions: 

(1)  Preproduction  Qualification  Tests  are  formal  contractual  tests  that  ensure 
design  integrity  over  the  specified  operational  and  environmental  range.  These  tests 
usually  use  prototype  or  preproduction  hardware  fabricated  to  the  proposed  production 
design  specifications  and  drawings.  Such  tests  include  contractual  reliability  and 
maintainability  demonstration  tests  required  prior  to  production  release.  At  a  minimum, 
embedded  diagnostics  capabilities  and  the  interfaces  to  external  diagnostic  elements 
should  be  tested  and  evaluated  during  preproduction  qualification  tests.  As  a  goal,  the 
capability  of  external  diagnostic  elements  should  also  be  tested  and  evaluated. 

(2)  Production  Qualification  Tests  ensure  the  effectiveness  of  the  manufacturing 
process,  equipment  and  procedures.  These  tests  are  conducted  on  a  sample  lot  taken  at 
random  from  the  first  production  lot,  and  are  repeated  as  the  process  or  design  is  changed 
significantly,  and  when  a  second  or  alternate  source  is  brought  on  line.  These  tests  are 
also  conducted  against  contractual  requirements.  The  utilization  of  diagnostic  resources 
in  the  manufacturing  process  and  the  requirement  for  capture  of  diagnostic  data  from  the 
manufacturing  process  should  be  evaluated  during  production  qualification  testing. 

The  completion  of  Preproduction  Qualification  Test  and  Evaluation  before 
Milestone  III  decisions  are  made  is  essential  and  will  be  a  critical  factor  in  assessing  the 
system’s  readiness  for  production. 

Operational  Test  and  Evaluation  (OT&E)  is  the  field  test,  under  realistic 
conditions,  of  any  item  (or  key  component)  of  weapons,  equipment  or  munitions  for  the 
purpose  of  determining  the  effectiveness  and  suitability  for  use  in  combat  by  typical 
military  users;  and  the  evaluation  of  the  results  of  such  tests.  Operational  testing  is 
accomplished  in  an  environment  as  operationally  realistic  as  possible.  The  entire 
diagnostic  capability  should  be  assessed  during  OT&E  as  well  as  the  integration  of  the 
diagnostic  capability. 
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The  Test  and  Evaluation  Master  Plan  (TEMP)  must  clearly  specify  development 
and  operational  test  events.  However,  DT&E  and  OT&E  are  not  necessarily  serial  phases 
in  the  evolution  of  a  weapon  system.  During  critical  acquisition  cycle  transitions,  elements 
of  DT&E  and  OT&E  may  be  combined  or  occur  in  parallel,  but  not  at  the  expense  of  either 
development  or  operational  test  realism  nor  before  sufficient  DT&E  can  reasonably  assure 
that  the  system  is  ready  to  enter  dedicated  operational  testing.  DT&E  may  continue  into 
the  Production  and  Deployment  Phase,  along  with  OT&E,  to  address  system 
enhancements,  correction  of  deficiencies,  or  modifications.  In  all  cases,  test  planning  for 
all  test  phases  must  be  addressed  in  the  system  TEMP. 

Test  and  evaluation  planning  is  initiated  at  the  inception  of  the  development 
process  to  ensure  adequate  planning,  programming  and  budgeting  of  test  resources  and 
to  facilitate  test  scheduling  to  support  major  program  decision  milestones.  Reliability 
assurance  should  be  well  underway  before  the  initiation  of  system  performance  tests. 
System  deficiencies  must  be  addressed  through  a  dynamic,  well-documented,  and  tightly 
managed  test-analyze-fix  and  retest  program.  The  evaluation  of  embedded  diagnostic 
elements  should  be  injected  into  these  reliability  assurance  tests. 

A  TEMP  is  required  for  all  major  defense  acquisition  programs.  The  TEMP 
defines  and  integrates  test  objectives,  critical  issues,  systems  characteristics, 
responsibilities,  resources  and  schedules  for  test  and  evaluation.  Test  resource 
requirements  must  be  addressed  in  the  TEMP,  along  with  a  critical  analysis  of  any 
shortfalls  that  will  impede  the  full  test  and  evaluation  of  the  system.  The  need  for  and  the 
availabilitv  of  the  various  diagnostic  elements  which  make  up  the  diagnostic  capability  is 
addressed  in  the  TEMP.  Plans  to  correct  existing  or  anticipated  test  resource  limitations 
are  also  included,  as  is  a  listing  of  evaluation  criteria  delineating  critical  parameters 
permitting  continuous  oversight  and  independent  assessment. 

DoD  5000.3-M-1  contains  the  guidelines  for  the  preparation  of  the  TEMP. 
Detailed  guidance  on  the  diagnostic  inputs  to  the  TEMP  are  orovided  below. 
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Detailed  Guidance  -  Diagnostic  Inputs  to  TEMP 
Part  I  -  System  Details 

Section  2b.  Interlaces 

Establish  diagnostics  as  a  system  that  is  required  to  accomplish  the  mission. 
Section  3  -  Required  Technical  Characteristics 

Include  diagnostic  performance  levels  in  the  listing  of  key  technical 
characteristics,  performance  goals  and  thresholds. 

Section  4  -  Required  Operational  Characteristics 

Include  diagnostics  performance  parameters  as  key  operational  effectiveness  and 
suitability  characteristics,  goals  and  thresholds. 

Part  II  -  Program  Summary 

Section  1  -  Management 

Identify  organizational  elements  responsible  for  diagnostics  T&E. 

Section  2  -  Integrated  Schedule 

Ensure  Diagnostics  T&E  are  Included  and  conform  to  System  T&E  schedule. 

Part  III  -  DT&E 

Define  in  detail  the  diagnostics-related  test  objectives  vor  DT&E.  Relate  those 
objectives  to  system  operational  concept. 

Section  t  -  Critical  Technical  Characteristics 

Discuss  the  availability  of  diagnostic  elements  as  ft  impacts  the  ability  to  evaluate 
the  total  diagnostic  capability.  Determine  the  criticality  of  the  availability  of  the 
diagnostic  elements.  Describe  diagnostic  workarounds  and  the  risks  (associated 
with  those  workarounds)  being  taken  by  not  being  able  to  evaluate  the  diagnostic 
capability  provided  by  the  integration  of  ail  of  the  diagnostic  elements. 

Section  2  -  DT&E  to  Date 

Summarize  diagnostics-related  DT&E  already  conducted. 

Describe  test  articles,  with  emphasis  on  how  they  differ  from  planned  production 
articles. 
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Section  4  -  Future  DT&E 

Discuss  aii  remaining  diagnostics  DT&E  planned,  including  description  of  system 
diagnostics  and  diagnostics  resources  to  be  utilized  In  DT&E.  Identify  how  these 
differ  from  the  production  unit/system  and  diagnostic  resources.  Define  detailed 
diagnostics  DT&E  objectives,  DT&E  events,  scope  of  testing  and  basic  scenarios. 
Relate  test  objectives  to  test  procedures. 

Part  IV  -  OT&E 

Discuss  all  planned  diagnostics-related  OT&E  activities  and  their  objectives. 
Include  planning  from  iOT&E  through  the  FOT&E  during  initial  production  and 
deployment  which  address: 

o  The  ability  of  the  diagnostic  capability  to  support  operational  effectiveness  and 
suitability 

o  Identification  of  diagnostic  deficiencies  in  the  production  system  (including 
deficiencies  in  aii  diagnostic  elements). 

Include  diagnostic  OT&E  considerations  in  the  following  sections. 

Section  1  -Critical  Operational  Issues 
Section  2  -OT&E  to  Date 
Section  3  -  Future  OT&E 

b.  OT&E  Objectives 

c.  OT&E  Events/Scope  of  Testing/Basic  Scenarios 

Part  V  -  Test  and  Evaluation  Resource  Summary 

Identify  the  key  resources  for  diagnostic  capability  DT&E,  OT&E  and  Production 
Acceptance  Test  and  Evaluation  (PAT&E)  that  are  unique  to  the  program. 

Section  1  -  Test  Articles 

Analyze  the  system  TEMP  to  Identify  the  actual  number  of  test  articles  planned 
for  DT&E,  OT&E  and  PAT&E  and  the  type  of  article  planned  (prototype,  pilot 
production,  production  articles).  Determine  If  this  is  sufficient  for  diagnostics- 
related  DT&E,  OT&E  and  PAT&E. 

Section  2  -  Test  Support  Equipment 

Briefly  describe  the  special  support  resources  (Instrumentation,  test  sites, 
facilities,  military  maintenance  manpower)  required  for  diagnostics  T&E,  and 
briefly  describe  the  steps  being  taken  to  acquire  them. 
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CHECKLIST 

O'  Have  diagnostics— related  inputs  to  the  TEMP 
been  prepared? 

or  Is  planning  for  diagnostics  T&E  consistent 
with  system-level  planning? 

Have  all  diagnostics-related  T&E  risks, 
critical  items  and  special  resource 
requirements  been  adequately  resolved? 

Does  planned  diagnostic  element  T&E  relate 
closely  to  the  actual  diagnostic  elements  to  be 
deployed  with  the  system? 

O'  Is  there  a  logical  relationship  between  diagnostic 

T&E  activities  and  the  Diagnostic  Maturation  Program 
(i.e.,  are  diagnostic  TdcE  results  going  to  be  used  as  a 
basis  for  the  maturation  of  the  diagnostic  capability)? 

Have  all  T<5cE  activities  been  analyzed  to  assess 
the  feasibility  of  evaluating  the  diagnostic  capability 
of  the  system  (e.g.,  can  Reliability  Growth  Testing 
contribute  any  diagnostic  related  T&E  information)? 
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PROD/ 

DEPL. 


DIAGNOSTICS 

DT&E 


DIAGNOSTIC  ACTIVITY 

Evaluate  diagnostics  performance  characteristics  during  Development  Test  and 
Evaluation  (DT&E)  activities  in  order  to  determine  diagnostic  capabilities  achieved  and  to 
identify  deficiencies  in  the  diagnostic  capability.  Diagnostics  DT&E  should  also  attend  to 
the  capability  achieved  by  the  integration  of  the  various  planned  diagnostic  elements 
(performance  monitors,  BIT/SIT,  testing:  automatic  and  manual,  maintenance  aids, 
technical  information  and  training  (skills))  into  a  comprehensive,  cohesive,  diagnostics 
subsystem. 

PROCEDURE 

Development  Test  and  Evaluation  is  the  T&E  conducted  by  the  contractor 
throughout  various  phases  of  the  acquisition  process  ‘i  ensure  the  acquisition  and  fielding 
of  an  effective  and  supportable  system  by  assisting  in  the  engineering  design  and 
development  and  verifying  attainment  of  technical  performance  specifications,  objectives 
and  supportability. 

Development  Test  and  Evaluation  also  includes  T&E  of  components, 
subsystems  and  preplanned  product  improvement  (P^l)  changes,  hardware-software 
integration  and  related  software,  as  well  as  qualification  and  production  acceptance 
testing.  Test  and  evaluation  of  compatibility  and  interoperability  with  existing  or  planned 
equipment  and  systems  is  emphasized.  Development  Test  and  Evaluation  encompasses 
the  use  of  models,  simulations,  and  testbeds,  as  well  as  prototypes  or  Full-Scale 
Development  models  of  the  system. 
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GUIDANCE 


The  thrust  of  the  Integrated  Diagnostics  Process  with  respect  to  DT&E  is  to 
include/inject  diagnostic  performance  evaluation  into  the  mainstream  of  DT&E  activities. 
This  is  done  such  that  diagnostic  performance  can  be  evaluated,  deficiencies  pinpointed, 
and  corrective  action  implemented  while  the  system  is  still  in  development. 

The  diagnostics  DT&E  effort  assists  the  diagnostic  design  and  development 
process,  and  verifies  attainment  of  diagnostic  technical  performance  specifications, 
requirements,  and  objectives.  As  such,  it  is  an  integral  part  of  the  weapon  system  design 
process.  Through  the  provision  of  diagnostics  DT&E  data,  there  is  a  feedback  reiterative 
loop  back  into  the  integrated  diagnostics  activities  in  process,  including  the  diagnostic 
system  engineering  analysis;  diagnostic  risk  analysis,  allocation  of  diagnostic  goals; 
diagnostic  trades  for  system  optimization;  diagnostic  design  trades;  and  the  identification 
and  performance  of  diagnostic  design  tasks.  Through  this  methodology,  the  diagnostic 
design  is  corrected,  improved,  or  updated,  and  the  diagnostic  design  matures. 

Sufficient  diagnostics  DT&E  must  be  accomplished  before  the  Milestone  III 
decision  to  proceed  to  production.  This  will  ensure  that  the  major  specified  diagnostics 
design  and  development  requirements  for  the  Full-Scale  Development  Phase  have  been 
met,  with  respect  to  performance  requirements  and  specifications  contained  in  program 
documents. 

The  Contractor  Program  Manager  should  be  as  actively  involved  in  diagnostics 
DT&E  to  ensure  that  valid  tests  are  being  performed,  valid  results  documented,  and  valid 
data  accumulated  and  to  ensure  that  a  closed-loop  analytic  approach  is  used  to  pinpoint 
and  correct  diagnostic  deficiencies.  The  Contractor  Program  Manager  should  also  ensure 
that  every  opportunity  is  being  taken  to  evaluate  diagnostics-related  parameters.  This 
may  involve  a  wide  range  of  test  activities,  including  reliability  tests,  performance  tests, 
human  factor  tests,  etc.  Basically  whenever  a  system,  subsystem  or  component  is  being 
operated,  it  is  subject  to  a  failure.  The  diagnostics  requirements  associated  with  dealing 
with  the  failure  should  be  viewed  as  an  opportunity  to  assess  the  diagnostic  capability. 

The  Contractor  Program  Manager  should  evaluate  the  results  and  data  from 
DT&E.  Based  upon  the  results  and  data,  critical  areas  should  be  identified.  Appropriate 
modifications  should  be  made  to  the  TEMP  with  respect  to  planning  for  OT&E  activities. 
Any  significant  deviations  from  the  Diagnostic  Maturation  Plan  and  attainment  of  specified 
diagnostic  capability  levels  should  be  tracked. 

The  Contractor  Program  Manager  must  determine,  based  upon  the  scope,  basis 
and  results  of  the  T&E  activity,  what  the  degree  of  confidence  is  that  the  deployed 
diagnostic  capability  will  achieve  operational  suitability  and  logistic  supportabllity 
requirements  of  the  system.  If  either  the  scope  of  testing,  the  basis  of  testing,  or  the 
results  of  testing  are  far  from  the  deployed  capability,  then  confidence  should  be  low,  and 
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diagnostics  should  be  identified  as  a  risk  item  in  the  TEMP.  Specific  efforts  should  be 
taken  to  resolve  or  reduce  this  risk.  Scope  of  testing  here  refers  to  the  evaluation  of  the 
diagnostic  capability  achieved  as  a  result  of  the  integration  of  diagnostic  elements  across 
system  assembly  levels  and  maintenance  levels.  Therefore,  the  full  scope  of  diagnostics 
T&E  is: 


The  scope  of  diagnostic  T&E  should  include  fault  detection  and  isolation 
accuracy  and  timeliness  provided  by  performance  monitoring,  BIT/SIT,  automatic  and 
manual  testing,  technical  information  and  maintenance  aids,  maintenance  procedures, 
manpower,  personnel  and  skill  levels  at  the  system,  subsystem,  LRU/LRM,  SRU  levels 
across  planned  maintenance  echelons  (Organizational,  Intermediate  and  Depot). 

Any  deviation  from  this  full  scope  of  T&E  means  that  full  confidence  cannot  be 
ascribed  to  the  planned  diagnostic  capability. 

These  factors  must  be  evaluated  in  terms  of  their  interrelationships  also.  For 
example,  the  Depot-level  test  capability  may  be  deemed  a  low-risk,  non-essential  portion 
of  the  diagnostics  capability.  Suppose  then  that  the  false  alarm  rate  proves  to  be  high  and 
the  ambiguity  resolution  proves  to  be  poor.  Soon,  the  Depot  capability  becomes  critical, 
with  large  numbers  of  UUTs  awaiting  test,  and  spares  supplies  being  depleted. 

"Basis  of  testing"  refers  to  the  extensiveness  of  test  and  the  procedures  utilized 
for  T&E.  The  basic  issue  is  whether  the  T&E  was  performed  such  that  confidence  can  be 
ascribed  to  the  results  of  the  test.  It  is  usually  unrealistic  to  plan  for  exhaustive  testing  of 
diagnostics  because  of  the  many  and  varied  failure  modes  to  which  the  system  is 
subjected.  Fault  simulations  and  analytic  models  are  often  used  to  evaluate  test 
effectiveness.  The  Contractor  Program  Manager  must  evaluate  the  meaningfulness  and 
the  realism  of  the  "basis  for  testing." 

Even  with  a  reasonably  large  sample  of  inserted  faults,  a  demonstration  can 
yield  only  limited  data  on  actual  test  effectiveness.  However,  a  demonstration  is  also 
useful  in  validating  some  of  the  assumptions  and  models  that  were  used  during  the  earlier 
testability  analysis,  and  prediction  efforts  which  were  based  upon  a  much  larger  fault  set. 
If  certain  assumptions  or  models  are  invalidated  by  the  demonstration  or  T&E  activity, 
appropriate  portions  of  test  effectiveness  predictions  and  analyses  should  be  repeated 
and  new  predictions  should  be  made. 

Diagnostics  DT&E  requirements  are  performed  in  accordance  with  the  System 

TEMP. 


The  major  approaches  of  DT&E  for  diagnostics  Include  actions  : 

o  To  proceed  in  phase  with  the  system  and  support  equipment  development, 
so  that  Built-In  Test  (BIT)  is  tested  and  evaluated  concurrently  with  system 
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performance;  BIT  and  System  Integration  Test  (SIT)  tested  and  evaluated 
concurrently  with  subsystem  integration  and  system  testing;  and,  system 
integration  and  safety  testing  are  concurrent  with  diagnostic  testing  of  BIT 
and  SIT  features. 

o  To  implement  with  the  Diagnostics  Maturation  Program  so  that  deficiencies, 
ambiguities,  and  additional  failure  modes  identified  during  DT&E  are 
recorded  in  a  timely  manner  to  ensure  traceability  and  appropriate 
corrections  are  made  to  the  integrated  diagnostic  procedures. 

o  To  evaluate  embedded  diagnostic  design  as  a  separate  entity  in  order  to 
assure  that  it  has  been  incorporated  adequately  as  part  of  the  system 
design. 

o  To  evaluate  for  100%  diagnostic  capability  in  selected  critical  areas  of 
system  design  using  fault  insertion  techniques. 

o  To  analyze  the  system  design  hierarchy  of  test  tolerances  (e.g.,  between 
system  BIT  and  LRU  and  SRU-level  BIT)  in  order  to  minimize  false  alarms. 

o  To  complete  feasibility  DT&E  on  prototype  and  preproduction  units  in  order 
to  assess  technical  risks  and  develop  solutions  to  remedy  deficiencies. 

During  FSD,  specific  diagnostic  capability  segments  of  DT&E  efforts  include  the 
following  requirements: 

o  When  available,  ATE  shall  be  evaluated  for  initial  use  supporting  build  and 
check-out  of  systems.  Manual  procedures  and  associated  operational 
prototypes  shall  be  developed  for  support  of  test  activities. 

o  Engineering  evaluation  of  the  diagnostic  elements  capability  at  subsystem 
and  system  levels  shall  be  conducted  in  concert  with  system  integration 
testing  activities,  including  evaluation  tests  in  the  engineering  laboratory  and 
system  integration  test  facilities. 

o  Effective  development  of  a  diagnostic  capability  requires  that  testing  of 
diagnostic  capabilities  proceed  concurrently  with  prime  and  support 
equipment  development  in  an  orderly  and  planned  time-phased  manner. 
The  object  of  the  following  diagnostics  testing  approach  is  to  provide  a  viable 
Organizational-  and  Intermediate-level  diagnostic  capability  for  use  in 
support  of  flight  and  operational  testing  activities  to  provide  for  early 
maturation  of  the  diagnostic  capability.  It  should  also  be  a  program  objective 
to  validate  the  diagnostic  capability,  as  well  as  initial  reliability  and 
maintainability  requirements  before  production. 
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o  During  early  equipment  development  tests,  built-in  test  features  should  be 
tested  and  evaluated  concurrently  with  equipment  performance  testing.  BIT 
performance  is  just  as  essential  to  overall  weapon  system  performance  as 
the  usually  emphasized  aspects  of  equipment  performance.  Simulated 
equipment  failures  should  be  used  to  assist  in  BIT  testing  and  evaluation. 

o  As  equipment  progresses  to  subsystem  integration  and  performance  testing, 
BIT  and  System  Integrated  Test  (SIT)  features  should  be  concurrently 
tested,  evaluated,  and  corrected.  Simulated  or  emulated  equipment  failures 
should  again  be  used  for  BIT/SIT  testing  and  evaluation. 

o  System  integration  and  safe-for-flight  testing  of  equipment  should  include 
diagnostic  testing  of  BIT  and  SIT  features  to  assure  readiness  of  this 
capability  for  Flight  Test  Support.  Concurrently,  Organizational-level  support 
equipment  required  for  diagnostic  support  should  be  tested  to  enable  its  use 
in  the  test  program,  together  with  Preliminary  Maintenance  Manuals  for 
initial  Operational  Test  and  Evaluation.  Simulation  of  equipment  failures  to 
evaluate  diagnostic  capabilities  should  be  included  in  this  testing  effort. 

o  Qualification  testing  of  both  prime  and  support  equipment  shall  include 
validation  of  diagnostic  capability,  which  is  a  required  aspect  of  both 
equipment  and  system  performance.  Simulated  equipment  failures  should 
be  included  in  the  diagnostic  validation  test  program.  Evaluation  of  BIT/SIT 
should  also  be  conducted  during  environmental  extreme  testing  of  the  prime 
equipment  and  support  equipment,  to  assure  its  proper  functioning 
throughout  the  required  equipment  performance  envelope. 
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CHECKLIST 


O'  Does  the  Integrated  Test  Plan  provide  adequate 

detail  concerning  specific  T&E  procedures,  data  bases, 
models,  test  articles  and  scope  of  testing? 

O'  Have  critical  or  high  risk  items  related  to  diagnostic 
capability  been  identified  and  highlighted? 

Are  the  necessary  test  articles  available  to 
conduct  realistic,  timely  tests? 

Is  the  level  of  government  involvement  in  diagnostics 
DT&E  adequate  to  ensure  validity  of  tests  performed, 
results  documented,  data  accumulated? 

E/  Is  there  a  direct  feedback  loop  in  the  engineering 

development  effort  to  deal  with  diagnostic  deficiencies? 

Has  every  opportunity  to  evaluate  diagnostics  during 
DT&E  activities  been  identified? 


e i  Are  diagnostics— related  DT&E  activities  consistent  with 
the  Diagnostics  Maturation  Plan? 
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WEAPON 
SYSTEM 
ACQ.  PHASE 

OPER. 

REQMTS. 

CONCEPT 

EXPLOR. 

DEM/ 

VAL 

FSD 

PROD/ 

DEPL. 

WEAPON 

SYSTEM 

ACTIVITIES 

IOT&E  FOT&E 

DIAGNOSTIC 

ACTIVITIES 

A  A 

DIAGNOSTICS 
OT&E  I 

DIAGNOSTIC  ACTIVITY 


Diagnostic  performance  characteristics  must  be  evaluated  in  a  realistic 
operational  environment  during  Operational  Test  and  Evaluation  (OT&E)  activities  in  order 
to  determine  diagnostic  capabilities  achieved  and  to  identify  deficiencies  in  the  diagnostic 
capability.  Diagnostics  OT&E  should  focus  on  the  capability  achieved  by  the  integration  of 
the  various  planned  diagnostic  elements  into  a  comprehensive,  cohesive  diagnostics 
subsystem.  The  Contractor  Program  Manager  should  provide  any  required  assistance. 

PROCEDURE 

Operational  Test  and  Evaluation  (OT&E)  is  the  field  test,  under  realistic 
conditions,  for  the  purpose  of  determining  the  effectiveness  and  suitability  of  the  system  or 
equipment  for  use  in  combat  by  typical  military  users;  and  the  evaluation  of  the  results  of 
such  tests. 

GUIDANCE 


Operational  Test  and  Evaluation  (OT&E)  activities  include  Initial  OT&E  (IOT&E) 
and  Follow-on  OT&E  (FOT&E).  The  results  of  DT&E  activities  should  be  analyzed  by  the 
Contractor  Program  Manager  to  ensure  consistency  and  continuity  of  T&E  activities. 
Operational  Test  and  Evaluation  (OT&E)  must  be  accomplished  by  a  separate  government 
facility  prior  to  the  Milestone  III  decision.  Diagnostics  OT&E  is  performed  to  provide  a  valid 
estimate  of  the  operational  effectiveness  and  suitability  of  the  system’s  integrated 
diagnostics  design  and  procedures  using  test  items  sufficiently  representative  of  the 
expected  production  items. 
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Major  approaches  to  diagnostics  OT&E  include: 
o  Testing  in  an  environment  as  operationally  realistic  as  possible 
o  OT&E  initiated  as  early  as  possible  during  the  FSD  Phase 
o  Testing  for  adherence  to  overall  OT&E  objectives,  with  respect  to  diagnostics 
o  Continued  coordination  with  the  Diagnostics  Maturation  Program 
o  Evaluation  for  100%  diagnostic  capability  in  selected  critical  areas 
o  Random  diagnostics  testing  in  noncritical  areas 

o  Further  analysis  of  test  tolerances  related  to  the  system  hierarchy  and 
embedded/external  diagnostic  procedures  in  order  to  minimize  false  alarms. 

Testing  (particularly  operational  tests)  and  data  collection  should  focus  on  the 
diagnostics  requirements.  Testing  and  data  collection  should  also  evaluate  the  specified 
parameters;  namely,  identification  of  critical  failures,  the  false  alarm  rate,  the  percentage  of 
faults  detected  and  isolated  automatically  or  manually,  associated  repair  times,  the 
unnecessary  removal  rate,  consistency  of  test  results,  and  the  adequacy  of  personnel 
skills  considering  all  maintenance  incidents. 

Use  of  the  diagnostic  capability  that  is  planned  for  field  maintenance  personnel 
should  be  required  whenever  there  is  a  need  for  system  maintenance.  This  applies  to 
maintenance  performed  by  either  the  contractor  or  the  user.  Thus  contractors  should  use 
the  diagnostic  capability  in  acceptance  and  qualification  tests  and  in  the  manufacturing 
and  quality  assurance  process  to  the  maximum  extent  possible.  In  addition  to  contributing 
to  the  maturation  of  the  diagnostic  capability,  it  is  anticipated  that  greater  contractor  use  of 
diagnostics  in  these  processes  could  result  in  production  cost  savings. 

The  diagnostic  capability  should  be  evaluated  with  respect  to  the  levels  planned 
and  set  forth  in  the  Diagnostics  Maturation  Plan.  Based  upon  the  difference  between 
planned  levels  of  capability  and  actual  levels  of  capability,  Diagnostics  T&E  and  corrective 
action  may  need  to  be  accelerated  or  adjusted.  (See  Requirement  #7.1  for  more 
information.) 

During  OT&E,  system  performance,  operational  suitability  and  supportability 
factors  are  evaluated  in  an  operationally  realistic  environment.  There  are  two  types  of 
information  that  can  be  obtained  for  Diagnostics  T&E:  1)  faults  within  the  system  and  how 
those  faults  were  identified  (diagnosed);  and,  2)  faults/deficiencies  within  the  diagnostic 
capability.  For  the  latter,  this  includes  evaluation  of  each  element  which  contributes  to  the 
total  diagnostic  capability,  as  well  as  the  capability,  achieved  by  integration  of  the 
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diagnostics  elements.  Focused,  detailed  T&E  activities  discussed  in  Requirement  #  6.2 
should  be  continued.  The  former  type  of  data  can  be  obtained  as  a  result  of  Reliability 
Growth  Testing.  The  following  specific  information  should  be  evaluated  for  each  fault 
occurrence. 

1 .  How  did  the  failure  manifest  itself? 

2.  Was  the  manifestation  due  to  stressing  of  the  system  beyond  normal 
operational  limits? 

3.  If  a  BIT  alarm  occurred,  was  it  the  result  of  a  confirmed  failure? 

4.  What  techniques  were  used  to  isolate  the  fault? 

5.  How  long  did  fault  isolation  take  using  those  techniques? 

6.  Was  the  failure  mission  or  operation  critical? 

7.  Was  it  a  new  or  unplanned  failure  mode?  Was  BIT  supposed  to  detect  the 
failure?  Did  it? 

8.  Is  this  failure  mode  expected  to  be  encountered  in  the  operational  system? 

9.  Should  provisions  be  included  in  the  diagnostic  capability  to  deal  with  this 
failure  mode? 

10.  Will  this  involve  a  modification/addition  to  BIT?  ATE?  Manual  Test 
Equipment?  Maintenance  Procedures?  Skill  Levels?  Technical  Data? 
Maintenance  Aids? 

11.  Is  an  ECP  required? 

12.  Is  further  investigation  required? 

If  yes  -  What  plans  have  been  made? 

If  no  -  Why  not?  (brief  description) 

13.  Is  correction  of  the  diagnostic  deficiency  part  of  contractual  requirements? 
Tied  to  incentive  or  warranty  provisions? 
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CHECKLIST 


Are  diagnostics  OT&E  test  articles  sufficiently  repre¬ 
sentative  of  the  expected  production  items? 

Is  the  diagnostics  OT&E  environment  as 
realistic  as  possible? 

Do  diagnostics  OT&E  plans  include  evaluation  for 
100%  diagnostic  capability  in  selected  critical  areas? 

Do  OT&E  plans  include  analysis  of  test 
tolerances  related  to  the  system  hierarchy 
and  off— line/on— line  diagnostics  procedures 
in  order  to  minimize  false  alarms? 

Of  Is  diagnostics  evaluation  included  in  a  broad 
spectrum  of  OT&E  activities 
(e.g.,  Reliability  Growth  Testing)? 

O'  Is  the  scope  of  diagnostics  OT&E  broad  enough 
to  realistically  do  a  preliminary  evaluation  of  the 
fielded  diagnostic  capability  provided  by  the 
combination  of  all  the  diagnostic  elements? 
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MATURATION  OF  THE  DIAGNOSTIC  CAPABILITY 
OVERVIEW 


Historically,  often  a  weapon  system’s  diagnostic 
capability  does  not  meet  its  performance 
requirements  prior  to  deployment.  The  basic 
reason  for  this  is  that  all  faults  cannot  be  predicted 
and,  thus,  adjustment  of  the  diagnostic  capability 
is  required  during  the  first  few  years  after 
deployment.  Essentially,  this  requires  a  well- 
planned  maturation  period,  which  allows  for  the 
growth  of  the  diagnostic  capability.  Closely 
coupled  with  this  maturation  is  the  requirement  for 
collection  and  analysis  of  data  relating  to  the 
performance  of  this  diagnostic  capability,  both  in 
the  field  and  in  the  factory.  Care  must  be 
exercised  by  both  the  government  and  the 
contractor  to  assure  that  proper  and  detailed  data 
is  collected.  Early  planning  for  this  maturation 
period  is  a  must. 


TESTABILITY/ 

DIAGNOSTICS 


PROGRAMMATIC 


REQUIREMENTS 


DESIGN 


ASSESSMENT 


REVIEWS 


EVALUATION 


IMPORTANT  CONSIDERATIONS  TO  BE  ADDRESSED 


Reqmt. 

7.1  Prepare  a  detailed  Diagnostics  Maturation  Plan  early  in  the  acquisition 
process. 

7.2  Determine  the  diagnostic  data  collection  requirement  and  establish  a 
system  for  collection  and  analysis. 


7-1 


MATURATION  PUNNING 


REQUIREMENT  #7.1 


WEAPON 
SYSTEM 
ACQ.  PHASE 

OPER. 

REQMTS. 

CONCEPT 

EXPLOR. 

DEM  / 
VAL 

FSD 

PROD/ 

DEPL. 

WEAPON 

SYSTEM 

ACTIVITIES 

£  £ 

SYSTEM  PDR 

SPEC. 

DIAGNOSTIC 

ACTIVITIES 

A  A 

PUN  UPDATE 

DIAGNOSTIC  ACTIVITY 


Most  diagnostic  implementations,  no  matter  how  well  conceived,  require  a 
period  of  time  for  identification  of  problems  and  corrective  action  to  reach  specified 
performance  levels.  This  requirement  is  established  in  order  to  formalize  the  diagnostics 
maturation  and  to  allow  the  maturation  to  be  initiated  early  in  the  test  and  evaluation 
process.  This  requirement  is  initiated  early  so  that  early  identification,  tracking,  and 
correction  of  diagnostic  problems  is  achieved.  The  planning  for  this  activity  is  formalized 
by  the  development  of  a  Diagnostic  Maturation  Plan  or  other  appropriate  document. 

PROCEDURE 


It  is  the  Contractor  Program  Managers  responsibility  to  prepare  the  Diagnostic 
Maturation  Plan. 

The  Contractor  Program  Manager  must  ensure  that  the  plan  Is: 

1.  Comprehensive 

o  Across  all  diagnostic  elements 
o  Includes  the  integration  of  the  elements 

2.  Timely 

o  Is  initiated  early  to  plan  for  the  required  resources  and  implement 
corrective  actions 

o  Maturation  is  completed  by  Milestone  IV,  per  DoD-lnstruction-5000.2 
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3.  Coordinated 

o  Includes  coordinated  activities  from  the  "ilities" 
o  Utilizes  standard  data  collection  systems 

4.  Cost  Effective 

o  Allows  data  collection  to  be  transferable  and  usable  by  government  (i.e., 
DT&E  and  production  test  data). 

GUIDANCE 


A  program  to  mature  the  diagnostic  capability  should  be  planned  for  the  early 
fielded  production  systems.  A  one-to-three-year  maturation  program  should  be  planned 
for  complex  weapon  systems  which  have  extensive  automatic  testing  capability.  For  major 
weapon  systems,  the  coordination  with  Milestone  IV,  Logistic  Readiness  and  Support 
Review  (DoD-lnstruction-5000.2)  is  essential.  This  program' should  include  provisions  for 
on-site  collection  of  diagnostic  performance  data,  with  engineering  follow-up  to  provide 
corrective  actions. 

The  plan  should  define  an  approach  and  methodology  to  assure  that  as 
development,  test  and  evaluation,  and  early  operational  use  of  the  system  progress, 
problems  presented  by  new  failure  modes,  test  voids,  ambiguities,  and  test  tolerance 
difficulties  are  recognized  and  defined,  and  their  solutions  are  traceable  to  diagnostic 
software  and  manual  procedure  updates.  The  plan  should  recognize  that  such 
occurrences  are  expected  and  normal  and,  therefore,  should  concentrate  on  problem 
recognition,  definition,  and  correction,  with  appropriate  tracking  for  traceability. 

The  approach  and  methodology  defined  should  recognize  that  a  basic  element 
of  the  integrated  diagnostics  concept  is  identification  of  the  set  of  faults  which  are  known 
or  expected  to  occur.  The  methodology  shall  provide  for  definition  of  this  set,  initially 
through  Failure  Modes  and  Effects  Analysis,  Testability  Analysis,  and  other  tools  and 
experience.  Provision  for  growth  of  this  set,  as  new  failure  modes  are  encountered  during 
testing  and  deployment,  should  be  incorporated  in  the  plan,  together  with  explicit  criteria  to 
be  used  in  deciding  whether  or  not  a  newly  encountered  fault  shall  be  added  to  the  set  of 
faults  for  which  explicit  diagnostic  procedures  (as  opposed  to  more  general  procedures) 
are  provided  for  detection  and  isolation  of  the  fault.  The  life  cycle  cost  effectiveness  of 
adding  explicit  diagnostic  procedures  for  the  newly  encountered  fault  shall  be  one  factor 
considered  in  the  decision. 

The  plan  should  provide  for  an  orderly  development  and  maturation  process  for 
diagnostic  software  and  manual  procedures  throughout  the  development,  test  and 
evaluation,  and  early  operational  use  time  periods  of  the  weapon  system  and  its 
subsystems.  Methodology  to  assure  timely  and  continuing  technical  support  for  this 
maturation  process  by  both  contractor  and  government  activities,  with  a  minimum  of 
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administrative  delays,  should  be  a  feature  of  the  plan.  Orderly  transition  of  technical 
responsibilities  from  the  contractor  to  the  government  should  also  be  addressed. 

The  plan  should  present  milestones  to  be  met.  This  will  assure  that  the  final 
system  achieves  the  required  degree  of  diagnostic  capability.  The  plan  shall  show  the 
time  phasing  of  each  task  and  its  interrelationship  with  other  tasks.  It  should  identify 
required  data  review,  verification,  and  utilization  to  accomplish  the  required  tasks  and  to 
report  progress,  problems,  and  tradeoffs.  The  plan  should  assure  the  proper 
implementation  of  diagnostic  design  features  by  designers  and  subcontractors. 

This  plan  will  enable  the  procuring  activity  to  monitor  and  evaluate  the 
contractor’s  progress  toward  achieving  the  required  diagnostic  capability  through  the 
system  design,  development,  test,  and  evaluation  process.  The  government  may 
establish  contractual  incentives  for  appropriate  milestones  throughout  the  diagnostic 
development,  test,  and  evaluation  process.  Milestones  selected  shall  include  completion 
of  design  for  testability  assessment  and  other  diagnostic  system  design  assessments; 
completion  of  diagnostic  test  element  and  diagnostic  system  evaluations,  in  concert  with 
equipment  design  evaluation  testing  at  the  LRU/subsystem  level;  and  diagnostic  system 
testing,  in  concert  with  systems  integration  test  facilities  and  during  the  operational  test 
program.  The  plan  should  also  provide  for  government  evaluation  and  final  acceptance  of 
the  automatic  test  programs  and  manual  troubleshooting  procedures  in  maintenance 
technical  publications. 

During  the  Dem/Val  Phase,  maturation  planning  is  centered  on  preliminary 
planning  for  data  collection,  analysis  and  coordination  with  similar  requirements  for 
reliability,  maintainability,  logistics,  data  collection,  analysis  systems,  etc.  Specifically,  this 
planning  should  identify  potential  data  sources,  such  as: 

o  Laboratory  testing 
o  Developmental  testing 
o  Operational  test  and  evaluation 
o  Acceptance  testing 
o  Preproduction  testing 
o  Production  testing 
o  Operation. 

The  requirement  for  diagnostic  data  collection  should  be  coordinated  with  similar 
requirements,  such  as: 

MIL-STD-785 

Task  104  -  Failure  Reporting  Analysis  and  Corrective  Action  (FRACAS) 

Task  105  -  Failure  Review  Board 

Task  301  -  Environmental  Stress  Screening 

Task  302  -  Reliability  Development/Growth  Test  (RDGT) 
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MIL-STD-470 

Task  104  -  Data  Collection,  Analysis,  and  Corrective  Action 
MIL-STD-471 

Maintainability  Verification/Demonstration/Evaluation 
MIL-STD-1 388-1 

Task  501  -  Supportability  Test,  Evaluation,  and  Verification 
MIL-STD-781 

Reliability  Testing  for  Engineering  Development,  Qualification  and  Production 
MIL-STD-21 55 

Failure  Reporting,  Analysis,  and  Corrective  Action  System 
MIL-STD-21 65 

Task  103  -  Testability  Data  Collection  and  Analysis  Planning 

No  standard  format  for  the  Diagnostics  Maturation  Plan  exists.  The  plan  may  be 
incorporated  in  another  plan,  such  as  the  SEMP.  The  guidance  provided  above  should  be 
completed  with  the  data  collection  and  feedback  system  (Requirement  #7.2)  established  in 
the  preparation  of  this  plan. 
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CHECKLIST 

E2f  Does  the  Diagnostic  Maturation  Plan  include  a  strategy 
for  the  collection  of  diagnostic  performance  data 
through  DT&E,  OT&E,  Production,  Initial  Operational 
Use,  and  Deployment? 

Ef  Are  the  data  sets  collected  throughout  the  above 
periods  realistic,  comprehensive,  and  timely?  Is  there 
a  logical  flow  through  the  data  sets  to  be  collected  so 
that,  as  maturation  proceeds,  data  from  one  period  can 
be  logically  used  in  the  next? 

□f  Does  the  plan  include  provisions  for  all  diagnostic 
elements  —  embedded  and  external  — 
as  well  as  the  integration  of  the  diagnostic  elements? 

Ef  Is  the  integration  of  the  diagnostic  elements  planned 
for  early  enough  to  allow  evaluation  and  cost-effective 
corrective  action  (e.g.,  prior  to  production  go-ahead)? 

E^  Do  the  data  sets  to  be  collected  allow  for 
government  capture  and  use  of  the  data? 

□f  Are  standard  government  data  collection  systems 
utilized? 

Ef  Is  there  an  appropriate  degree  of  coordination  with 
similar  requirements  (e.g.,  if  Reliability  Growth 
Testing  is  a  program  requirement,  is  data  from  that 
testing  planned  to  be  coordinated  with  Diagnostics 
Maturation)? 

Ef  Are  plans  and  provisions  for  dealing  with 

subcontractors  and  vendors  in  the  Diagnostics 
Maturotion  Plan? 

E "t  Are  configuration  control  requirements  included  in  the 
maturation  planning  for  the  prime  system,  as  well  as 
for  the  diagnostic  elements? 
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CHECKLIST  (cont.) 

E2f  Docs  Maturation  Planning  include  provisions  for  both: 

1.  Adequacy  of  the  diagnostic  elements,  with  respect 
to  the  specified  allocated  capability,  and 

2.  Unplanned  failure  modes,  which  may  arise  throughout 
OT&E,  DT4cE,  Production  Test,  and  Field  Use  Test 

e/  Is  a  structured,  closed-loop,  analytic  process  planned 
for  the  resolution  of  any/all  deficiencies? 

Ear  Is  it  clearly  laid  out  who  is  responsible  for  what 
throughout  the  various  periods  of  the  maturation 
process  (i.e.,  who  (government  or  contractor)  is 
responsible  for:  (1)  data  collection;  (2)  analysis; 
ana  (3)  corrective  action  through  DTdet,  OT&E,  Produc¬ 
tion  Test  and  Field  Use  Test). 
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LECTION  ACTION 

DIAGNOSTIC  ACTIVITY 


Data  relating  to  the  performance  and  effectiveness  of  the  diagnostic  capability 
must  be  collected  during  development,  production,  and  operation.  This  data  is  used  as 
the  basis  for  the  evaluation  of  diagnostics  and  for  the  correction  of  deficiencies. 

The  key  thrust  of  this  activity  is  definition  of  appropriate  data  to  be  collected, 
maximum  use  of  data  collected,  coordination  of  data  collection  systems,  and  a  structured 
approach  to  corrective  action. 

PROCEDURE 


The  Contractor  Program  Manager  is  responsible  for  the  implementation  of 
diagnostic  data  collection  and  feedback  requirements.  This  includes  development  and 
implementation  of  a  cradle-to-grave  system  for  both  contractor  and  government  use. 

The  earlier  diagnostic  performance  deficiencies  are  Identified,  the  sooner  a 
more  cost-effective  solution  can  be  implemented.  Therefore,  diagnostic  data  collection 
and  feedback  is  initiated  early  in  the  test  and  evaluation  process,  continues  through 
production  test,  and  extends  into  the  operational  environment.  Throughout  these  phases, 
different  types  of  data  are  collected,  different  data  collection  procedures  and 
methodologies  are  used,  and  different  types  of  analysis  technique  are  conducted. 
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GUIDANCE 


There  are  no  standard  methods  for  data  collection  and  analysis.  As  indicated 
under  Requirement  #7.1,  Maturation  Planning,  the  collection  of  this  type  of  data  is 
controlled  by  a  number  of  military  standards.  The  requirements  for  the  standards  which 
deal  with  logistics,  reliability,  maintainability,  testability,  human  engineering,  and  safety 
overlap  one  another  (many  times  data  required  by  one  may,  indeed,  be  required  by  the 
other(s)).  Thus  close  coordination  among  these  various  data  requirements  is  needed.  A 
single  data  base  is  desirable. 

In  addition,  these  data  systems  are  required  during  system  development,  as  well 
as  after  the  system  is  deployed.  There  must  be  compatibility  between  the  contractor’s  data 
system  and  the  follow-on  government  data  system,  so  that  traceability  exists  from  cradle- 
to-grave. 


The  data  collection  procedures  closely  follow  the  test  and  evaluation  functions. 
As  explained  in  DoD  Directive  5000.3,  Test  and  Evaluation,  the  time  periods  and 
sequences  for  Development  Test  and  Evaluation  and  Operational  Test  and  Evaluation 
vary  from  program-to-program.  They  can  overlap  and  even  be  done  as  a  combined  test 
and  evaluation.  Thus  there  are  no  standard  guidelines  that  specify  the  exact  points  in  the 
weapon  system  acquisition  phase  where  data  is  to  be  collected.  The  system  must  be 
flexible  to  incorporate  data  as  data  is  generated. 

The  Contractor  Program  Manager  should  ensure  that  the  proper  data  is 
collected  and  that  corrective  actions  are  pursued.  Care  must  be  taken  to  collect  only  that 
data  required  to  assure  that  the  diagnostic  capabilities  are  performing  as  required. 
Automated  data  collection  systems  can  be  employed.  Usually  these  are  more  effective,  as 
they  are  less  dependent  on  human  motivation  to  supply  the  required  information. 

Corrective  analysis  and  actions  should  be  in  a  closed-loop  system,  so  that  each 
deficiency  identified  remains  an  open  item  until  it  is  formally  documented  as  being 
corrected. 


The  data  collection  and  feedback  system  should  be  designed  so  that  specific 
information  is  collected  on  the  performance  of  the  entire  diagnostic  capability,  as  well  as 
for  each  of  the  diagnostic  elements  that  make  up  the  diagnostic  capability.  The 
information  must  be  collected  in  quantitative  form,  if  possible,  and  related  to  System 
Specification  requirements.  Thus  the  following  guidelines  on  the  type  of  data  to  be 
collected  need  to  be  tailored  so  that  the  information  can  be  related  to  System  Specification 
requirements  and  so  that  it  is  clearly  apparent  who  is  to  supply  the  information  and  when 
this  information  is  to  be  supplied.  Examples  of  the  type  of  data  to  be  collected  follow. 
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Diagnostic  Data  Feedback 

o  Effectiveness  and  efficiency  of  oach  diagnostic  element 

o  Effectiveness  and  efficiency  of  the  diagnostic  elements  as  an  integrated 
system 

o  Operational/support  impact  of  the  diagnostic  deficiencies 
o  Corrective  action(s)  which  should  be  taken  or  have  been  taken. 

BIT  Effectiveness 

o  Fault  isolation  time. 

Tracking  of  False  Alarms 
o  Type  of  alarm 

o  Frequency  of  alarm  occurrence 
o  Cause  (if  known) 

o  Potential  consequences  of  ignoring  the  alarm  (crew  safety,  mission 
reliability) 

o  Operational  costs  of  responding  to  false  alarms  (aborted  missions,  degraded 
mode  operation,  system  down  time) 

o  Support  costs  associated  with  the  false  alarm 

o  Operational  environment  when  alarm  occurred. 

ATE  Effectiveness  Feedback 

o  Workarounds  required  to  overcome  mechanical  or  electrical  deficiencies  in 
the  UUT/ATE  interface 

o  Consistency  of  ATE  FD  results  with  initial  BIT  indications 
o  Repeatability  of  ATE  test  results 
o  Ambiguity  size 
o  Fault  isolation  time. 
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Integration  of  Diagnostic  Elements 

o  Consistency  of  diagnostic  resources  with  the  training/skill  levels  of  assigned 
personnel? 

o  Effect  of  false  alarms  and  unnecessary  removals  on  operational  availability 
and  maintenance  workload 

o  Shop  throughput 

o  Adequacy  of  technical  information 

o  Logistic  delay  time 

o  BIT  reliability 

o  ATE  reliability. 


Diagnostic  data  collection  and  diagnostic  capability  performance  assessment 
may  lead  to  the  requirement  for  corrective  action.  Corrective  action  may  involve  redesign 
of  prime  equipment,  test  equipment,  interface  devices,  maintenance  documentation,  built- 
in  test  circuits,  diagnostic  software,  and  ATE  test  programs.  All  changes  must  be  made 
under  strict  configuration  control. 
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CHECKLIST 

E/  Has  the  data  collection  system  been  designed  to  collect 
only  the  required  data?  In  quantitative  terms? 

E ~i  Are  the  government,  contractor,  and  subcontractor  data 
collection  systems  compatible? 

E/  Is  there  direct  communication  back  and  forth  between 
the  person  who  reports  a  problem  and  the  person  who  is 
responsible  for  correcting  the  problem? 

O'  Will  all  known  failures  be  reported? 

Of  Will  all  failures  be  analyzed  to  sufficient  depth  to  iden¬ 
tify  failure  causes  and  necessary  corrective  actions? 

Ef  Will  all  failure  analysis  reports  be  closed  out  within 
30  days  of  failure  occurrence  or  rationale  provided  for 
any  extensions? 

O'  Will  corporate  management  be  automatically  alerted  to 
failures  exceeding  close-out  criteria? 
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LESSONS  LEARNED 
DESIGNING  A  NEW  SYSTEM 


I.  INTRODUCTION 

A  young  Air  Force  technician  assigned  to  maintain  one  of  today’s  sophisticated 
weapon  systems  is  on  his  way  to  another  day  of  work.  As  he  travels  his  prescribed  route, 
he  reflects  on  the  intricacies  of  the  equipment  with  which  he  works.  He  sighs  in 
amazement  regarding  how  easily  and  accurately  he  knows  what  to  replace  when  things  go 
wrong. 


Shortly  after  arriving  at  his  duty  station,  he  enters  his  code  at  a  computer 
terminal  and  is  provided  with  a  work  order  for  his  first  task  of  the  day.  The  work  order 
concerns  a  malfunction  which  was  detected  during  a  flight  completed  just  one-half  hour 
earlier.  A  quick  glance  at  the  work  order  reveals  which  system  failed,  what  time  it 
occurred,  and  the  Line  Replaceable  Unit  (LRU)  which  is  to  be  replaced  to  correct  the 
problem.  After  a  quick  trip  to  a  supply  point  for  a  serviceable  LRU,  with  tool  box  and 
checklist  in  hand,  he  departs  for  the  flight  line.  The  defective  LRU  is  changed  within 
minutes  after  his  arrival.  A  quick  operational  check,  using  the  checklist  and  on-board  test 
system,  confirms  that  no  other  failures  have  occurred,  and  the  system  is  declared 
operational. 

Back  at  the  Intermediate  shop,  the  flight  line  portion  of  the  work  order  is  closed. 
This  is  quickly  done,  with  a  minimal  amount  of  information  input  at  the  computer  terminal 
regarding  the  work  accomplished.  The  defective  LRU  is  placed  on  Its  corresponding 
automatic  test  equipment  (ATE).  Keys  within  the  LRU  provide  identification  information  to 
the  computer  contained  within  the  ATE.  Failure  conditions  and  symptoms  recorded  on- 
aircraft  at  the  time  of  the  failure  are  also  transferred  to  the  ATE  computer  via  the  computer 
network.  The  ATE  rapidly  goes  through  a  set  of  tests  specifically  tailored  to  the  reported 
failure  conditions  and  the  failed  single  component  is  identified. 

The  failed  component  is  replaced  with  a  new  component  obtained  from  the 
bench  stock  located  within  the  shop.  After  being  checked  again  with  the  ATE  to  verify 
serviceability,  the  LRU  is  given  a  quality  control  inspection  and  returned  to  the  supply 
point,  where  it  once  again  becomes  a  serviceable  asset. 

The  above  scenario  (or  parts  thereof)  has  been  a  goal  of  military  services  for 
many  years.  Great  strides  have  been  made  in  diagnostics  toward  its  achievement,  yet 
even  total  success  in  limited  areas  does  not  lie  immediately  at  hand. 
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The  reasons  that  success  is  fleeting  are  many.  They  include  budget 
constraints,  a  relative  lack  of  importance,  political  considerations,  time,  and  the  complexity 
of  the  task-just  to  mention  a  few.  This  appendix  presents  a  few  glimpses  of  diagnostic 
activity  on  recent  programs,  results  obtained,  and  lessons  teamed. 

The  information  presented  is  a  composite  of  program  experiences  derived  major 
aircraft  systems,  as  well  as,  strategic  missile  systems. 

A.  Contractor  Program  Manager  Concerns 

Experience  has  shown  that  Contractor  Program  Manager  difficulties  in  obtaining 
good  diagnostic?  He  in  two  main  areas.  The  first  is  succeeding  within  man-derived 
constraints,  which  limit  the  emphasis,  the  amount  of  resources  which  can  be  applied,  and 
program  efficiency  related  to  diagnostics.  The  second  major  area  of  concern  is  the 
technical  difficulties  involved.  Specifically  for  the  Contractor  Program  Manager,  this 
relates  to  the  development  of  a  clear  understanding  of  diagnostics  requirements  and 
assuring  their  achievement  in  the  hardware/support  system  design. 

Both  of  the  above  areas  are  demonstrated  in  the  following  pages  of  this 
document. 

B.  Time  Frame 

This  appendix  presents  lessons  learned  information  from  all  phases  of  an 
equipment’s  life  cycle,  In  consonance  with  the  scope  of  the  Program  Managers  Guide,  to 
which  it  is  attached.  As  such,  it  describes  activities  which  may  have  taken  place  over  the 
last  20  years,  or  more,  even  though  we  are  mainly  addressing  recently  deployed  weapon 
systems. 

C.  Constraints,  I.  eM  Time,  Money,  and  Political 

The  lessons  learned  described  in  this  document,  in  almost  every  instance,  have 
something  to  do  with  trying  to  succeed  within  constraints  imposed  on  the  program  due  to 
limited  resources  and  other  political  considerations.  One  major  aircraft  program  poses  an 
excellent  example  of  the  usual  limited  resource  problems,  compounded  by  political 
decisions  not  in  the  best  interest  of  the  program.  The  original  development  effort  was 
accomplished  from  1971  through  1977,  at  which  time  the  program  was  canceled.  Four- 
and-one-half  years  later,  in  March  1 982,  the  program  was  reinstated.  In  order  to  make  up 
for  lost  time,  a  concurrent  Full-Scale  Development  and  Production  Program  was 
contracted.  The  initial  expectations  were  that  the  earlier  effort  would  allow  for  this 
concurrence.  The  reality  was  that  there  were  hardware  changes  and  other  development 
efforts  required  which,  if  time  had  permitted  orior  to  production,  would  have  resulted  in 
fewer  production  problems  and  better  diagnostics  at  first  deployment. 
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II.  INTERPRETING/ENFORCING  REQUIREMENTS 

What  is  specified  in  the  procurement  specification  and  the  contractual  Statement 
of  Work  is  what  the  government  expects  to  receive.  In  the  area  of  diagnostics,  the 
government  experience  on  past  programs  has  not  been  the  best.  Systems  have  been 
introduced  into  the  inventory  with  diagnostics  that  have  proven  to  be  incomplete,  unable  to 
test  to  the  desired  level,  or  simply  do  not  operate  as  advertised.  The  basic  foundation 
upon  which  all  other  successes  or  failures  are  directly  dependent  is  the  clear 
understanding  of  the  actual  requirements. 

A.  Need  Statements  and  Work  Statements 

All  of  the  programs  surveyed  in  the  preparation  of  these  documents  seem  to 
have  an  item  in  common  dealing  with  their  diagnostic  requirements.  That  commonality 
factor  is  that  the  quantitative  diagnostic  requirements  imposed  are  derived  without  a  great 
deal  of  thought  and  analysis.  Typically,  diagnostic  requirements  are  more  what  has  been 
judged  by  someone  to  be  realistic  values,  rather  than  a  product  of  the  various  studies 
performed  to  determine  these  requirements. 

DoD-lnstruction  5000.2  and  other  related  documents  describe  a  structured 
acquisition  process  beginning  with,  among  other  things,  the  development  of  a  Mission 
Area  Analysis  and  a  Mission  Need  Statement.  Included  in  the  Mission  Need  Statement  is 
a  discussion  of  the  Mission  and  Threat, 'Alternative  Concepts,  and  Technology 
involvement.  Subsequently,  during  the  Concept  Exploration  Phase,  studies  are  conducted 
to  develop  a  System  Concept  Paper  which  more  thoroughly  defines  possible  alternatives, 
and  a  selected  concept.  Many  items  are  taken  under  consideration  during  this  time  frame 
including  readiness,  maintainability,  manpower  and  training. 

It  is  this  process  which  generally  drives  the  development  of  the  procurer-tint 
specifications.  These  functions  are  primarily  the  concern  of  Contract  Program  Manager 
however  inputs  are  sometimes  requested  from  the  contractor.  Failure  to  consider 
testability  when  providing  these  inputs  may  limit  chances  for  successful  diagnostics 
implementation  later  in  the  program.  Overall  diagnostic  and  testability  in  general  should 
be  given  more  concern  at  this  early  stage  of  development. 

B.  Cost-Effective  Requirements 

Specifications  sometime  create  requirements  either  greater  or  less  than  those 
actually  required  to  meet  the  needs  of  diagnostics.  This  is  very  expensive,  from  a 
taxpayer’s  point  of  view,  considering  that  excessive  requirements  increase  costs  in  every 
acquisition  phase,  i.e.,  development,  procurement,  and  support.  Specifications  which 
impose  values  less  than  those  required  to  fulfill  the  basic  need  of  diagnostics,  cause 
excessive  costs,  mainly  in  the  operational  phase.  These  come  from  developing  work 
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arounds,  not  being  able  to  fulfill  mission  needs,  and  driving  more  assets  into  the  repair  and 
support  system. 

MIL-STD-1388-1A  (Logistics  Support  Analysis)  describes  specific  tasks  to  be 
performed  to  develop  those  considerations.  These  tasks  include  a  use  study,  a 
comparative  analysis  with  existing  systems,  consideration  of  standardization,  investigating 
technological  opportunities,  and  trade-off  analysis.  This  process  is  geared  to  the 
development  of  specifications  which  is  driven  by  inputs  performed  under  the  previous 
contract  phase. 

This  Standard,  issued  in  1 983,  was  not  available  to  be  imposed  on  any  of  the 
programs  covered  by  this  appendix.  However,  judging  from  programs  developed  using 
the  MIL-STD-1388-1A  process,  it  is  a  consensus,  that  improved  diagnostics  requirements 
and  program  plans  would  have  resulted  had  this  process  been  available,  the  Contractor 
Program  Manager  plays  a  key  role  in  this  process  by  ensuring  that  tasks  performed  under 
this  standard  are  completed  and  submitted  in  a  timely  manner.  It  is  an  iterative  process  of 
definition,  synthesis,  trade-offs,  test,  and  evaluation  which  influences  requirements  for  the 
next  phase  of  the  contract.  A  good  point  to  remember  is  that  it  is  very  difficult  for  the 
Contractor  Program  Manager  to  proceed  to  the  next  phase  while  waiting  on  inputs  from 
the  present  phase. 

C.  Specification  Interpretation 

The  proper  diagnostic  specifications  necessary  to  meet  the  mission  need  is  one 
thing.  Describing  them  in  such  a  way  that  they  will  be  interpreted  properly  is  another. 

The  following  is  one  of  the  diagnostic  requirements  imposed  on  the  aircraft 
developer.  The  On-Board  Test  System  shall  provide  an  assurance  of  95%  to  the  aircrew 
that  the  system  performance  is  operationally  acceptable  or  that  the  indicated  failure  is 
valid  during  in-flight  performance  and  ground  readiness  tests.  It  shall  provide  fault 
isolation  to  an  LRU,  with  a  certainty  of  at  least  75%  in  the  ground  fault  isolation  mode." 

Another  requirement  stated  that  "false  alarms  could  not  exceed  2%."  Both 
seemingly  good  requirements,  but  two  problems  ensued  in  their  accomplishment.  First 
and  foremost  was  the  problem  in  the  definition  of  the  percent  base.  Percentages  are  often 
used  in  defining  requirements.  But  when  so  used,  it  must  be  stated  as  a  percent  of  what. 
False  alarms,  as  a  percent  of  the  possible  alarms,  give  one  result.  False  alarms,  as  a 
percent  of  the  number  of  total  alarms  indicated,  give  another.  When  written,  one  must 
assume  that  achievement  based  on  the  definition  of  the  writer  would  meet  mission  needs. 
In  reality,  when  achieved  based  on  a  legal  and  implied  definition,  the  results  were  far  from 
those  required  by  the  operational  command. 
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A  second  problem,  but  in  this  case  a  lesser  problem,  was  a  conflict  between  the 
requirements.  The  first  requirement  above  allows  a  5%  false  alarm  rate  (100  minus  the 
95%  accuracy).  The  second  allows  only  2%.  Specification  ambiguity  leads  to 
interpretation  which  will  not  necessarily  end  with  the  desired  result. 

III.  STRUCTURING  DESIGN  CONCEPTS/CONSTRAINTS 

A.  Controlling  vs.  Constraining  the  Contractor 

Today’s  trend  in  specification  and  contractor  direction  is  to  provide  the 
contractor  with  the  maximum  leeway  in  meeting  specified  requirements.  The  objective  is 
to  allow  the  contractor  to  define  alternatives,  select  from  the  alternatives  those  which  he 
can  best  implement,  and  provide  a  product  which  meets  all  of  the  requirements. 

Existing  systems  covered  by  this  document  were  all  developed  under  a  more 
structured  specification  approach.  The  previous  school  of  thought  was,  generally 
speaking,  that  the  more  things  which  can  be  controlled  by  the  specifications,  the  more 
chance  the  end  product  will  be  produced  as  desired.  Experience  with  that  approach  has 
led  to  the  more  open  trend.  This  is  because  the  tighter  approach  did  not  allow  the 
contractor  to  make  maximum  use  of  his  many  possible  alternatives. 

Good,  easy,  and  frequent  communications  between  the  customer  and 
contractor  diagnostics  personnel  is  a  must.  A  specific  item,  noted  almost  unanimously 
with  personnel,  is  the  importance  of  the  function  provided  on  that  program  by  devoted 
diagnostic  managers  in  both  the  Air  Force  and  customer  organizations.  The  shortfall, 
however,  was  that  these  managers  did  change  from  time-to-time.  On  long  programs  this 
is  to  be  expected.  The  lesson  to  Contractor  Program  Managers  is  to  provide  for  these 
changes.  Close  coordination  with  potential  backups  and  well-documented  decisions  will 
minimize  upsets  due  to  personnel  changes. 

Another  important  point  noted  was  that  these  diagnostic  managers  need  some 
very  special  skills  besides  those  of  a  skillful  manager.  Each  should  be  an  experienced 
engineer,  with  a  thorough  understanding  of  the  hardware  involved,  diagnostic  methods, 
and  support  concepts. 

B.  Establishing  and  Designing  to  the  Maintenance  Concept 

The  logistic  support  analysis  tasks  of  MIL-STD-1 388-1  which  are  concerned  with 
the  development  of  maintenance  concepts  and  constraints  are  very  important  for  the 
diagnostics  community.  The  Contractor  Program  Manager  should  ensure  that  these  tasks 
are  completed,  as  appropriate,  and  that  the  resulting  maintenance  concept  is  well 
understood  by  all.  The  MIL-STD-1 388-1  tasks  are  structurwd  to  ensure  consideration  of 
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existing  resources,  compatibility  with  deployment  and  operational  requirements,  and  the 
use  of  trade  studies. 

The  tie  between  the  diagnostic  method  and  the  maintenance  concept  is 
bidirectional.  They  need  to  be  established  in  unison.  The  maintenance  concept  is 
developed  based  on  expected  diagnostic  capabilities.  The  diagnostic  design  ultimately 
forces  the  real  maintenance  concept. 

Established  Air  Force  maintenance  policies  generally  utilize  system  operation  as 
the  final  determinant  of  the  need  for  maintenance,  if  the  system  is  functioning  within 
tolerance,  don’t  fix  it.  A  unique  situation  has  developed  on  the  aircraft.  Due  to 
redundancies  designed  in  the  systems,  overall  operation  appears  to  be  normal,  while 
some  specific  parts  thereof  are  not  functioning  as  they  should.  The  diagnostics  says 
these  parts  should  be  replaced.  System  operation  appears  to  indicate  everything  is  OK. 
To  date,  partially  due  to  the  lack  of  confidence  in  the  aircraft-diagnostics  and  partially  due 
to  established  habits,  often  these  type  malfunctions  are  not  being  repaired.  Generally  the 
diagnostics  is  believed  to  be  faulty  and  no  maintenance  action  is  taken  until  something 
else  happens  to  make  the  system  inoperative. 

This  experience  shows  that  changing  existing  practices  is  slow.  If  it  is  confused 
with  the  lack  of  confidence  in  the  diagnostics,  the  change  is  even  more  difficult. 

The  logistics  manager,  ATE  manager,  and  automatic  test  equipment  designer 
are  all  vital  elements  in  determining  what  off-equipment  testing  is  required.  Once  the 
option  for  automated  testing  is  confirmed,  the  ATE  designers  must  convince  the  Unit 
Under  Test  (UUT)  designer  to  incorporate  "design  to"  criteria  for  maintainability,  reliability, 
and  testability.  Care  must  be  taken  to  define  the  need  for  ATE,  how  the  ATE  is  to  be 
used,  how  the  UUT  will  be  designed  for  built-in  test,  and  interfacing  abilities.  ATE 
effectiveness  is  directly  and  immediately  dependent  on  this  co-development  with  the  UUT. 


IV.  Meaningful  Prediction  and  Assessment  Methodology 

In-process  assessment  of  diagnostics  achievements  has,  in  the  past,  been  less 
than  adequate.  In  fact,  one  of  the  most  definitive  and  often  repeated  lessons  is  the  need 
for  an  operational  period  to  mature  the  diagnostic  design.  That  lesson  is  described  in 
paragraph  VII  below.  Prediction  and  assessment  techniques  have,  in  the  past,  failed  to 
provide  sufficient  information  to  uncover  all  of  the  inadequacies  and  shortcomings. 
Significant  emphasis  is  currently  being  placed  on  testability  analysis,  reliability,  and 
maintainability  assessment  tools  under  the  umbrella  of  Computer-Aided  Acquisition  and 
Logistics  Support  (CALS).  With  that  emphasis,  one  should  expect  great  improvements  in 
assessment  techniques.  The  point  for  the  Contractor  Program  Manager  on  this  subject  is 
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to  ensure  that  predictions  and  assessments  performed  are  sufficient  to  demonstrate  that 
diagnostic  requirements  are  being  fulfilled. 

A.  Methodology 

The  CALS  initiative  would  include  diagnostics  design  as  an  integral  part  of  the 
CAE/CAD  design.  The  concept  is  that  rules  and  techniques  would  be  established  in  the 
computer  workstation.  As  a  specific  item  is  designed,  it  is  constantly  checked  for  test 
access,  built-in  test  capability,  and  other  established  rules. 

This  concept  works  fine  for  evaluating  the  diagnostic  characteristics  of  a  single 
electronic  assembly.  Evaluation  of  a  weapon  system’s  on  board  test  system  is  another 
question.  For  the  above  aircraft,  a  complete  integration  lab  was  developed  to  test  the 
diagnostics  software  in  a  functional  environment.  That  process  was  useful,  but  still  under 
the  best  of  lab  conditions  some  things  cannot  be  developed  to  the  optimum  level.  An 
excellent  example  is  the  philosophy  for  checking  the  thrust  of  a  jet  engine.  Simulated  lab 
conditions  equate  more  to  an  aircraft  being  on  the  ground  where  thrust  is  compared  to  a 
reference  schedule  of  gross  thrust  versus  turbine  blade  temperature  at  two  discrete 
operating  points.  These  two  operating  points  are  the  intermediate  and  maximum  power 
setting.  To  develop  an  in-flight  thrust  check,  a  reference  has  to  be  calculated  to  monitor 
performance  across  the  entire  power  range.  This  reference  is  obtained  by  comparing  the 
engines  in  synchronization  to  one  another  in  flight.  This  reference  requirement,  plus  many 
preconditions  necessary  for  calculating  or  examining  thrust,  dictates  actual  flight  testing  for 
development  of  a  valid  check. 

B.  Review  and  Feedback  Structure 

Time  and  management  emphasis  are  both  needed  to  assure  that  the  design 
benefits  from  the  assessment  process.  Logically,  one  does  not  need  a  whole  lot  of 
experience  to  understand  this.  However,  it  was  proved  once  again  on  the  aircraft. 
Concurrent  Full-Scale  Development  and  Production  meant  that  funding  for  studies  and 
analysis  occurred  so  late  that  the  results  could  not  be  implemented. 

Management  direction  is  also  needed  or  results  will  go  unheeded.  If  the 
decision  is  between  getting  an  aircraft  into  the  air  on  schedule  or  improving  the  diagnostic 
capability,  what  will  the  Program  Manager’s  decision  be? 

V.  DESIGN  REVIEWS 

Formal  Design  Reviews  provide  the  opportunity  for  the  customer  to  accept  what 
the  contractor  is  offering  or  insist  on  improvements.  If  the  contractor  can  demonstrate  that 
he  is  meeting  the  specifications,  the  customer  can  ask  no  more.  At  this  point,  it  is  the  role 
of  the  Contractor  Program  Manager  to  assure  that  sufficient  analysis  has  been  performed, 
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prior  to  design  reviews,  which  demonstrates  with  a  high  degree  of  confidence  that 
diagnostic  requirements  are  being  met. 

A.  Scheduling 

It’s  either  too  early  or  too  late.  Picking  the  optimum  time  for  reviews  is  very 
important.  Reviews  need  to  be  conducted  after  the  design  is  sufficiently  defined  to  make 
the  evaluation  (the  type  of  evaluation  being  determined  by  the  type  of  review),  but  before  it 
is  too  late  to  make  changes. 

The  only  identified  lesson  learned  from  experience  is  that  the  scheduling  for 
formal  reviews  is  typically  performed  at  the  beginning  of  the  program.  The  stage  of  the 
design  for  the  review  is  then  whatever  it  is  at  the  scheduled  time.  This  is  not  necessarily 
bad,  because  typically  the  designers  work  toward  having  a  reviewable  product  on  the 
established  schedule.  Usually,  reviews  cannot  be  delayed  without  jeopardizing  delivery 
schedules. 

B.  Format 

Diagnostics  usually  involve  the  "whole"  picture.  This  should  be  considered  in 
developing  the  review  agenda.  Prior  to  the  diagnostics  portion  of  the  review,  a  full 
understanding  of  the  hardware/software  design  and  maintenance  concept  is  required. 

C.  Contractor  Emphasis 

Messages  are  sent  to  the  contractor  telling  him  where  he  should  put  his 
emphasis,  based  on  the  importance  an  item  has  in  the  review,  if  the  Government  program 
Manager  and  his  review  team  place  little  emphasis  on  diagnostics,  the  contractor  gets  that 
message  and  typically  follows  suit.  A  sure  fire  way  to  hinder  the  progress  of  the  diagnostic 
design  team  is  to  indicate  to  the  Contractor  Program  Manager  that  diagnostics  are  "not 
important."  This  has  often  been  done  unintentionally,  in  the  past,  by  quickly  passing  over 
the  subject  in  the  review,  or  otherwise  indicating  a  minimal  concern.  The  Contractor 
Program  Manager  must  emphasize  the  importance  of  diagnostics,  especially  in  case  the 
Government  Review  Team  has  placed  little  emphasis  on  the  subject. 

VI.  Demonstrations 

Demonstrations  are,  in  general,  another  form  of  a  formal  review.  Thus  most  of 
the  points  made  in  the  previous  section  also  apply  here. 
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A.  Timeliness 

The  opportune  time  for  final  demonstration  of  diagnostics  does  not  exist,  if  a 
purpose  of  the  demonstration  is  to  identify  corrective  actions.  Efforts  to  schedule 
demonstrations  early  enough  to  minimize  the  impact  of  "failure"  have,  in  the  past,  resulted 
in  the  simulation  of  too  many  conditions  and  resources.  To  perform  a  complete 
diagnostics  demonstration,  all  operational  diagnostics  tools  must  be  in  place.  This 
includes  support  equipment--if  appropriate-training,  technical  publications,  and  any  other 
applicable  diagnostic  tool.  Attempts  to  simulate  or  work  around  the  absence  of  these 
operational  items  does  not  provide  for  a  complete  demonstration. 

B.  Simulated  vs.  Operational  Conditions 

The  problem  can  be  demonstrated  by  experience  with  a  recent  modification 
program  on  a  fighter  Attack  Radar.  The  modification  was  major-mainly  made  to  improve 
reliability  and  maintainability.  One  significant  portion  of  the  modification  was  the  rework  of 
the  built-in  test  (BIT)  capabilities. 

The  design  job  seemed  to  be  done  very  well.  Design  Reviews  were  passed. 
Demonstrations  of  the  new  BIT  performance  in  the  laboratory  exceeded  the  specifications 
and  expectations.  All  looked  like  a  job  well  done,  and  the  contract  was  considered 
complete. 


The  problem  was  that  on  the  aircraft,  in  operational  conditions,  the  BIT  does  not 
do  so  well.  The  BIT  serves  two  functions,  one  being  to  advise  the  aircrew  if  the  selected 
mode  is  operational,  the  other  serving  as  a  diagnostics  aid  to  maintenance  personnel. 
The  aircrew  function  performs  well,  which  is  not  surprising,  being  part  of  the  basic 
operational  requirement.  However,  the  diagnostics  portion  of  the  software  used  in  the  fault 
isolation  process  has  required  extensive  rework.  At  first  glance,  one  is  led  to  believe  that 
the  simulated  and  operational  conditions  must  differ  greatly.  This  being  the  case,  how 
does  one  explain  that  problems  reported  during  field  operations  can  later  be  demonstrated 
under  laboratory  conditions?  Simulated  conditions  may  in  fact  create  problems  at  times, 
but  like  in  this  case,  why  do  they  generally  appear  in  the  area  of  diagnostics?  Performing 
demonstrations  with  the  primary  objective  of  showing  operational  requirements  are  being 
fulfilled,  with  diagnostics  given  secondary  concern  only  delays  finding  problems  in  that 
area.  An  important  point  to  remember  is  that  diagnostics  must  be  given  equal 
consideration  to  operational  requirements  and  the  Demonstration  Phase  is  another 
chance  to  identify  this  problem. 
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C.  Providing  for  Resources 

Scheduling/obtaining  resources  for  the  demonstration  is  an  early  program 
function.  This  requirement  has  often  been  overlooked  or  minimized  in  the  past. 
Contractor  Program  Managers  need  to  be  fully  aware  of  the  demonstration 
plan/requirements  and  assure  that  they  are  included  in  the  very  earliest  top-level  planning 
documents. 

VII.  MATURATION 

Maturation  is  a  phase  which  has  been  identified  as  necessary  primarily  during 
development  of  new  systems/technology  for  the  embedded  and  external  diagnostic 
capability.  One  especially  critical  area  for  these  systems  is  the  inherent  requirement  for 
testing  under  actual  operating  conditions.  Maturation  becomes  necessary  to  refine  test 
method/fault  limits/diagnostics  logic  embedded  within  the  diagnostic  software  programs 
that  operate  these  systems.  The  predicted  operating  characteristics  of  the  various  on¬ 
board  systems  must  be  compared  to  the  operation  of  these  systems  as  they  interface  with 
other  systems  and  during  varying  environmental  conditions. 

A.  Early  Planning 

The  Contractor  Program  Manager  lesson  demonstrated  is  to  plan  for 
considerable  time  and  resources  to  allow  maturation.  The  original  development  plan  was 
to  mature  the  diagnostic  system  on  70  FSD  flights.  That  would,  it  was  thought,  provide  a 
mature  system  at  the  time  of  the  first  deployment  to  an  Air  Force  Main  Operating  Base. 
Early  in  the  Full-Scale  Development  Phase,  it  became  evident  that  that  plan  would  not  be 
sufficient.  A  new  plan  was  developed  to  use  468  sorties  over  the  years  1985  and  1986. 
The  wing  did  not  fly  the  required  number  of  sorties  over  that  time  period  and  the  program 
was  extended  through  November  1987.  Additional  aircraft  deliveries  and  an  increase  in 
sortie  generation  rate  produced  a  total  of  1 060  sorties  by  the  end  of  that  period.  With  that 
number  of  sorties,  sufficient  data  had  been  gathered  to  indicate  a  more-than-acceptable 
level  of  performance.  At  this  point,  it  is  estimated  that  as  a  general  rule,  at  least  400  to 
500  sorties  will  be  required  to  mature  an  on-board  test  system. 

B.  Operational  or  Flight  Test  Environment 

How  does  one  plan  for  500  sorties  prior  to  production?  Is  a  plan  to  fly  four  FSD 
aircraft  on  the  average  of  once  every  three  calendar  days  for  a  year  reasonable?  Is  a 
limited  production  block  appropriate  for  maturation?  These  are  questions  for  which  the 
Contractor  Program  Manager  must  get  answers  early  in  program  planning. 

Experience  has  identified  one  additional  consideration  to  be  included  in  making 
these  planning  decisions.  That  consideration  is  the  impact  a  partially  working  diagnostic 
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system  has  on  the  maintenance  technician.  If  technicians  lose  confidence  in  a  diagnostic 
aid,  they  will  not  use  it.  Further,  it  is  hard  to  convince  them  that  the  item  has  been 
improved  and  that  now  they  can  have  confidence  in  it.  Many  maintenance  technicians, 
who  have  been  exposed  to  inaccurate  diagnostic  methods,  have  never  been  convinced  to 
use  an  "improved"  version.  All  operating  bases  may  have  the  same  current  version  of  the 
diagnostic  systems.  Field  data  shows,  however,  that  the  bases  exposed  to  the  earliest 
and  poorest  version  continue  to  have  the  highest  false  alarm  and  cannot  duplicate 
maintenance  rates.  This  is  due  to  the  lack  of  trust  still  carried  from  the  early  experience. 
Thus  it  is  important  to  arrange  for  maturation  away  from  the  majority  of  operational 
technicians,  if  possible. 

C.  Implementing  Maintenance  Concept 

Special  training  will  need  to  be  provided,  if  the  maintenance  concept  utilizing  the 
planned  diagnostics  is  significantly  different  from  that  with  which  the  established  technician 
is  familiar.  Trends  are  also  in  place  today  to  isolate  to  and  replace  modules  on  the  aircraft 
rather  than  the  large  "boxes"  of  the  past.  Utilizing  the  diagnostic  indication  produced 
during  the  flight,  without  further  ground  verification,  is  also  a  current  trend.  Each  of  these 
"new"  concepts  must  be  thoroughly  understood  by  the  technicians,  so  that  the  maturation 
results  are  consistent  with  the  planned  fielded  maintenance  concept. 

VIII.  SUMMARY 

Diagnostics  must  be  a  simple  matter,  since  everything  always  works  at  the  end 
of  the  program.  However,  this  is  not  the  case,  and  the  perfect  situation  portrayed  in  the 
Introduction  has  yet  to  be  achieved.  Instead  of  the  capability  to  identify  the  one  LRU, 
often  the  ambiguity  group  is  as  much  as  four  LRUs.  The  ATE  which  can  isolate  the  failure 
to  a  single  failed  component  would  be  the  ideal  solution,  but  more  likely  than  not,  it  will 
only  be  to  one  or  sometimes  several  shop  replaceable  units  (SRUs)  or  a  particular  group 
of  SRUs.  The  steps  covered  here  are  only  some  of  the  very  basic  ones  required  to  ensure 
good  diagnostics.  However,  looking  at  many  different  programs,  one  finds  even  these 
simple  steps  have  been  omitted,  or  perhaps  accomplished  at  a  time  too  late  to  have  the 
desired  results.  The  reasons  are  many:  poor  communication  of  needs  or  goals,  time 
frame  restrictions,  money,  and  failure  to  properly  consider  the  importance  of  diagnostics. 
To  ensure  diagnostics,  it  must  be  addressed  at  all  phases  and  be  given  equal  importance 
to  other  performance  requirements.  If  the  system  cannot  be  maintained,  it  can  never  meet 
its  operational  requirements. 

Many  people  were  queried  in  the  development  of  this  document.  Most 
expressed  very  similar  lessons  learned. 
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CHECKLIST 

□f  Does  the  contractor  have  one  specific  person 
responsible  for  managing  all  the  diagnostic 
activities? 

□f  Does  the  person  filling  the  above  position  have 
a  good  understanding  of  logistics,  system  design, 
and  diagnostics  over  and  above  those  skills 
required  of  a  good  manager? 

o'  Have  procedures  been  established  within  the 
organization  to  facilitate  communication  on  a 
frequent  basis  among  the  individuals  involved 
in  diagnostic  design? 

Ef  Is  the  timing  for  the  various  diagnostic  studies 
and  analyses  adequate  for  assuring  that  results 
can  influence  the  diagnostic  design? 

Have  proper  priorities  been  demonstrated  at  all 
levels  of  management  on  both  the  government  and 
industry  sides  to  demonstrate  that  the  management 
of  the  diagnostic  design  is  really  an  important 
function? 

E/  Are  the  diagnostic  specifications  well  defined  and 
represent  exactly  what  is  needed? 

O'  Has  sufficient  time  been  allocated  for  the  maturation 
of  the  diagnostic  capability? 
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I.  INTRODUCTION 

This  guide  stresses  the  need  to  "design-in”  the  weapon's  diagnostic  capability 
as  the  design  of  the  prime  hardware  and  software  progresses.  The  concept  being  "do  it 
right  the  first  time."  Unfortunately,  the  possibility  of  addressing  the  design  of  the  diagnostic 
capability  beginning  at  Concept  Exploration  and  following  through  to  Deployment  is 
becoming  less  and  less  possible.  New  starts  are  becoming  less  frequent  and 
modifications  to  existing  systems  more  prevalent.  In  addition,  retrofits  to  existing  weapon 
system  diagnostic  capabilities  will  continue  to  be  required.  The  capability  to  design  or 
improve  the  diagnostic  capability,  beginning  at  any  point  in  the  acquisition  or  deployment 
of  a  weapon  system,  is  a  must.  Thus,  the  following  example  of  the  M-1  Abrams  tank. 

Deficiencies  in  the  diagnostic  capability  of  the  M-1  led  to  the  initiation  of  an 
Integrated  Diagnostics  Improvement  Program  which  is  a  positive  example  for  improving 
the  diagnostic  capability  after-the-fact.  As  with  the  M-1 ,  dissatisfaction  with  a  fielded 
diagnostic  capability  is  not  limited  to  just  the  user.  It  becomes  a  technology  issue,  a  cost 
issue,  and  certainly  a  political  issue.  In  the  case  of  the  M-1 ,  the  maintenance  issue 
became  a  national  issue,  draped  in  negatives. 
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To  develop  and  implement  the  necessary  improvements,  a  Joint  Working  Group 
for  Integrated  Diagnostic  Improvements  was  formed  with  a  charter  to  develop  a  system 
engineering  approach  to  the  Improvement  and  integration  of  Its  diagnostic  capability.  For 
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an  existing  system  with  diagnostic  problems,  redesign  of  the  system  or  subsystems  to 
include  testability  is  a  potential  solution.  Other  potential  solution  areas  include 
improvement  of  the  fielded  diagnostic  elements,  such  as  test  equipment,  test  programs, 
test  procedures,  technical  information,  maintenance  aids,  maintenance  personnel  skill 
levels,  etc.  The  costs  and  benefits  of  these  potential  solutions  must  be  carefully  analyzed. 
Integration  among  the  diagnostic  elements  also  must  be  maintained  or  achieved. 


II.  SUMMARY  OF  DIAGNOSTIC  PROBLEMS 

The  first  order  of  business  for  the  Joint  Working  Group  (JWG)  was  the 
development  of  integrated  diagnostic  improvement  objectives  and  development  of  a 
roadmap  to  achieve  the  objectives.  The  JWG  objectives  are  listed  below: 

o  Resolve  diagnostic  problems  related  to  support  of  Abrams 

o  Identify  cost-effective  solutions  to  the  problem  (s)  that  complement  short-term 
fixes 

o  Ensure  that  the  diagnostic  concept  supports  future  Air  Land  Battle  Doctrine 

o  Communicate  lessons  learned  to  combat/material  developers. 

The  JWG  defined  their  scope  of  effort  to  include  the  man,  test  equipment, 
technical  information,  training  and  tank  interfaces  (for  embedded  and  nonembedded 
diagnostics). 

The  JWG  developed  a  list  of  Abrams  problems  related  to  diagnostics.  The  JWG 
categorized  problems  according  to  embedded  and  nonembedded  diagnostic  elements. 
Prioritized  embedded  diagnostic  problems  are  presented  below.  Limited  Built-In  Test  (BIT) 
for  the  entire  vehicle  is  the  most  serious  embedded  diagnostic  deficiency.  The  extent  of 
the  BIT  deficiency  is  depicted  in  Figure  8,  M-1  Tank  Built-In  Test.  The  mobility  system 
does  not  have  an  embedded  diagnostic  capability.  For  the  remainder  of  the  tank 
electronic  systems,  on-line  fault  detection  coverage  is  only  46  percent.  The  majority  of  BIT 
coverage  fault  isolates  to  subsystem  functional  ambiguity  groups.  Embedded  off-line  fault 
isolation  coverage  includes  22  LRUs,  but  only  fault  isolates  to  the  single  LRU  27  percent 
of  the  time.  BIT  fault  isolation  to  the  single  LRU  is  limited  to  the  Laser  Range  Finder  and 
the  Thermal  Imaging  Unit.  In  most  cases,  nonembedded  diagnostics  are  still  required  to 
fault  isolate  to  a  single  LRU. 
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FIGURE  8.  M-1  Tank  Built-In  Taat 
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Diagnostic  problems  are  presented  below.  A  common  misconception  is  that  the 
Simplified  Test  Equipment  (STE)  core  hardware  is  the  root  of  all  diagnostic  problems.  The 
real  culprit  is  tank  diagnostic  design  which  contains  over  100  unique  cable  connectors 
resulting  in  the  proliferation  of  STE  Test  Program  Set  (TPS)  cables  and  connectors. 

ABRAMS  DIAGNOSTIC  PROBLEM  SUMMARY 

EMBEDDED  (TANK) 

1 .  Limited  BIT  for  entire  vehicle 

a. No  embedded  diagnostics  for  mobility  system 

b. No  BIT  for  sensors 

2.  Not  an  integrated  diagnostic  system 

3.  No  standard  diagnostic  connector  assemblies 

4.  Cannot  fault  isolate  to  a  single  LRU  or  cable  (Non-intrusively) 

5.  No  data  recording  capability 

a.  Intermittent  faults 

b.  Historical 

6.  Poor  LRU,  cable  and  connector  accessibility 

7.  No  standard  test  connector  design 

8.  Limited  embedded  sensors  for  vehicle  transmission 

9.  Limited  space  for  new  hardware  * 


NONEMBEDDED  DIAGNOSTICS 
STE  M-1  (specific  to  the  M-1) 

Lack  of  Effectiveness  in  the  Operational  Environment 
STE  -  CORE  (Generic) 

1 .  Does  not  meet  environmental  requirements 

2.  Display  not  visible  in  direct  sunlight 

3.  Seh-test  takes  too  long 

STE  -  TEST  PROGRAM  SETS 

1 .  Requires  cumbersome  cable  connections  (bulky) 

2.  Technical  manuals  not  written  for  use  as  TPS  documentation  with  STE 
(independent  action) 

3.  Test  Strategy 

a.  Examines  failure  from  component  level  rather  than  system  level 
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b. Stops  after  first  fault  identified 

c. Restart  to  recover  from  operator  error 

d. Experienced  mechanic  cannot  eliminate  test  steps 

e. Does  not  tak6  advantage  of  known  fault  data  (e.g.,  from  BIT) 

4.  Test  times  do  not  support  doctrinal  limits 

5.  Test  messages  subject  to  misinterpretation 

6.  Test  measurements  not  provided  to  mechanic 

TECHNICAL  MANUALS 

1.  Not  user  friendly 

a. Volume 

b.  Excessive  cross  referencing 

2.  Serial  symptomatic  approach  to  diagnostics  cumbersome  and  time 
consuming 

3.  Paper  technical  manuals  not  suited  for  field  use 

4.  Written  only  to  lowest  skill  level 

5.  Time  consuming  to  make  changes/difficult  to  control  process 

TRAINING/PERSONNEL 

1 .  Mechanics  lack  theoretical  knowledge  base  (back  to  basics) 

2.  Mechanics  lack  sufficient  advanced  and  sustainment  training 


III.  THE  SYSTEM  ENGINEERING  APPROACH  TO  ABRAMS 

INTEGRATED  DIAGNOSTICS  IMPROVEMENT 

In  addition  to  developing  a  list  of  Abrams  diagnostic  problems,  the  JWG 
identified  24  diagnostic  improvement  programs.  The  question  the  JWG  had  to 
answer  was:  If  the  problems  are  well  known  and  improvement  programs  are  under 
development,  why  is  there  still  a  diagnostic  problem  and  what  can  the  JWG  do  that 
has  not  been  done  before? 

The  answer  to  the  question  involves  approaching  diagnostic 
improvement  from  a  systems  engineering  and  integrated  diagnostics  perspective. 
The  JWG  discovered  that,  within  Army  Materiel  Command  (AMC),  existing 
diagnostic  improvement  programs  focused  on  different  problem  areas.  Some 
improvement  programs  covered  the  same  problem  areas,  other  improvement 
programs  did  not  adequately  address  any  problem  area.  The  JWG  determined  that 
if  they  were  to  add  anything  to  what  has  already  been  done,  a  systematic  approach 
to  the  problem  must  be  defined  and  implemented.  Their  approach  to  integrating 
diagnostic  programs  focuses  on  treating  the  diagnostic  elements  and  their 
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interfaces  as  a  system  from  the  unit  through  the  intermediate  levels  of 
maintenance. 


To  provide  a  comprehensive  and  cost-effective  plan  for  improving  the 
Abrams  diagnostic  capability,  the  Group,  using  a  systems  engineering  approach, 
developed  the  integrated  diagnostic  methodology  shown  in  Figure  9.  Based  on 
operational  requirements,  the  Group  prioritized  the  tank's  systems  and  subsystems 
at  the  different  levels  of  repair  and  prioritized  the  tank’s  systems  and  subsystems 
from  a  criticality  of  repair  point  of  view  at  the  different  levels  of  repair. 


DEVELOP  DIAGNOSTICS  REQUIREMENTS  BASED  ON 

-  OPERATIONAL  REOI IIREMENTS 

-  cRrricAi.mr  of  components  repair 

-  COMPONENTS  REPAIR  ECHELON 


oNrr 

-  develop  embedded  and  non  EM  bedded  diagnostic 

ALTERNATIVES  REQUIRED  TO  MEET  DIAGNOSFIC 
REQUIREMENTS 

-  CAN  PRESENT  DIAGNOSTIC  APPROACH  BE  IMPROVED  1 

•  DETERMINE  ACQISTTIUN,  LOGIS1 1C  AND  SUPPORT  COST 
ASSOCIATED  WH1I  EACH  ALTERNAOVE 

.  CONDUCT  TRADE  OFF  ANALYSIS  A  .RISK  ASSESSMENTOF 
ALTERNATIVES 

•  SELECT  OPTIMUM  DIAGNOSTIC  APPROACH 


INTERMEDIATE 

-  DETERMINE  DSESTS  MANPOWER/ 
WORKLOAD  ADDITIONS 

•  REQUIREMENTS  OF  F.D/TIS  ADDITIONS 

-  DETERMINE  CAUSES  OF  HIGH  NEOF 
RAT  ES 

-  ENVIRONMENTAL 

•  TOUiRANCE  CONE 

•  SWING  MAINTENANCE 

•  CARIES 

•  RECOMMEND  SOLUTIONS 


V 


DIVISION  SUPPORT 
AREA 


BRIGADE  SUPPORT 
AREA 


^  COLL  FT.  N 


BATTALION  TASK  FORCE 
SUPPORT  AREA 


DEVEI.OP  EMBEDDED 
DIAGNOSTICS 
CAPABn.TTTY  TO  MEET 
STATUS  REPORT  ING 
REQUIREMENTS 


COMPANY  TEAM 
SUPPORT  AREA 


FIGURE  9.  Integrated  Diagnostic  Methodology 
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The  JWG  segregated  diagnostic  improvement  into  two  phases;  short¬ 
term  fixes  and  long-term  solutions.  This  was  necessary  to  adjust  recommendations 
to  the  Abrams  production  schedule. 

Short-term  fix  recommendations  are  limited  to  ongoing,  government  and 
industry,  funded  embedded  and  nonembedded  diagnostic  programs  achievable 
through  1990. 

The  JWG  recognized  that  short-term  fixes  could  not  possibly  address  all 
of  the  Abrams  diagnostic  problems.  The  Group  also  recognized  that  certain  design 
problems  could  not  be  cost-effectively  remedied  in  the  near  term,  i.e.,  no  standard 
test  connector  design,  limited  embedded  sensors  for  vehicle  transmission  and  poor 
LRU,  cable  and  connector  accessibility.  These  types  of  problems  will  be  addressed 
in  the  long-term  solutions  plan. 

The  Group  chose  a  user  perspective  to  narrow  the  set  of  Abrams 
diagnostic  problems  to  a  manageable  subset  for  short-term  solutions.  From  a  users 
perspective,  what  needs  improvement  the  most  (non-prioritized)  includes: 

o  Provide  embedded  diagnostic  capability  for  mobility  system:  In  a 
battle  zone,  an  immobile  tank  is  a  dead  tank.  The  ability  to  rejoin  the 
battle  or  limp  home  is  impaired  by  the  fault  detection  and  fault 
isolation  time  required,  and  by  the  vulnerability  and  limited  quantity  of 
maintenance  teams  available  to  support  a  tank  company.  Embedded 
diagnostics  in  the  tank  which  can  detect  and  isolate  a  mobility  fault 
without  outside  intervention  reduces  diagnostic  time  to  the  minimum 
and  provides  each  tank  with  its  own  "built-in  diagnostic  team.” 

o  Reduce  size  and  weight  of  technical  manuals  and  Test, 
Measurement,  and  Diagnostic  Equipment  (TMDE):  An  M-1 13  or  M-88 
does  not  have  the  capability  to  carry  mechanics,  spare  parts,  cases  of 
test  equipment,  cases  of  cables  and  connectors,  and  12  feet  of 
technical  manuals.  A  tank  at  the  down  site  cannot  be  repaired  without 
mechanics  and  spare  parts.  The  only  candidates  left  for  size  and 
weight  reduction  are  the  technical  manuals  and  TMDE. 

o  Reduce  dependency  on  technical  manuals  and  STE:  In  the  Abrams 
diagnostic  history,  field  workarounds  were  adopted  to  circumvent  use 
of  the  STE  and  technical  manuals.  The  dedicated  contractor  support 
mechanics  did  not  depend  on  technical  manuals  and  the  STE.  They 
were  able  to  diagnose  faults  based  on  their  expertise  with  the  aid  of 
simple  test  equipment,  system  functional  flow  charts  and  wiring 
diagrams.  The  senior  NCOs  and  Warrant  Officers  observing  these 
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improvements  were  convinced  there  are  other  approaches  to 
diagnostics.  This  observation  led  to  the  creation  of  the  Master 
Diagnostician  (Master  "D")  concept.  The  Master  "D"  troubleshooting 
approach  reduces  the  dependency  on  technical  manuals  and  the 
STE,  allowing  the  maintenance  teams  to  carry  more  spare  parts.  It 
also  reduces  the  problem  of  trying  to  reference  a  paper  manual  in 
inclement  weather  and  reduces  diagnostic  time  by  eliminating  the 
requirement  to  connect  bulky  test  cables  and  connectors  to  the  tank. 

o  Reduce  requirements  for  personnel  possessing  Master  ”D"  skills 
(expert  system  instead):  The  Master  "D"  is  capable  of  troubleshooting 
the  tank  with  limited  technical  information  and  test  equipment.  The 
Master  "D"  relies  on  a  superior  knowledge  of  operational  theory  and 
fault  isolation  procedures.  The  problem  with  Master  "D"  is  the  cost  of 
training,  limited  availability  of  candidate  mechanics  and  low  retention 
of  personnel  with  Master  "D"  skills.  Given  the  current  state  of  expert 
system  technology,  the  logical  solution  to  the  Master  ”D"  problem  is  to 
incorporate  his  knowledge  of  the  tank  and  troubleshooting  procedures 
into  an  expert  system.  A  "Master  D  in  a  box"  eliminates  the 
requirements  for  a  mechanic  with  Master  "D"  skills,  reduces  training 
requirements  and  enhances  the  average  mechanics  skill  level  to  that 
of  a  Master  "D". 

o  Improve  speed  of  fault  diagnosis:  In  a  remove  and  replace 
maintenance  scenario  practiced  at  the  downsite  and  collection  point 
for  electrical  and  electronic  components,  fault  diagnosis  is  the 
bottleneck  to  system  repair.  Improvements  in  fault  detection  and  fault 
isolation  offer  the  greatest  payback  in  reducing  the  unit-level 
maintenance  burden.  Enhanced  speed  and  ease  of  diagnostics  at 
the  down  site  and  battalion  collection  point  will  discourage  use  of 
swing  maintenance  practices.  Arbitrary  removal  and  replacement 
causes  unacceptably  high  No-Evi^ence-of-Failure  (NEOF)  rates.  It 
also  increases  the  workload  o-<  the  battalion  maintenance 
organization  and  depletes  the  P  scribed  Load  List/  Authorized 
Stockage  List  (PLL/ASL)  inventory. 


IV.  THE  ABRAMS  DIAGNOSTIC  SYSTEM 

Ordnance  Center  and  Armor  School  JWG  representatives  rank  the  battle 
criticality  of  tank  subsystems  as  follows: 
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1.  Mobility 

2.  Firepower 

3.  Surveillance 

4.  Survivability 

5.  Communications. 

The  JWG  endorsed/recommended  short-range  programs  are  presented 
in  Figure  1 0.  Diagnostic  improvement  efforts  concentrated  on  the  top  three 
subsystems.  Mobility  is  covered  under  Hull  Embedded  Capability.  Firepower  and 
Surveillance  are  covered  under  Turrent  Embedded  Capability.  All  three 
subsystems  are  covered  under  Hull  and  Turrent  Embedded  Capability  Expansion 
and  Nonembedded  Capability. 

From  a  diagnostic  point  of  view,  embedded  and  nonembedded  diagnostic 
capabilities  are  all  encompassing.  Embedded  diagnostics  Includes  Built-In  Test 
(BIT),  Built-In  Test  Equipment  (BITE),  and  fault  data  recording  functions  to  support 
prognostics.  Recommended  short-range  embedded  programs  are  retrofitable  to 
existing  Abrams  tanks.  With  the  addition  of  an  optional  1553  multiplex  data  bus, 
short-range  embedded  programs  are  upwardly  compatible  with  Vetronics 
architecture  requirements. 

Nonembedded  diagnostic  recommendations  improve  current 
troubleshooting  procedures  and  lead  to  the  development  of  expert  system 
technology  to  improve  the  average  mechanics  troubleshooting  capability. 
Recommended  changes  to  the  nonembedded  test  strategy  decrease  diagnostic 
time  and  limit  the  use  of  cumbersome  test  equipment.  Nonembedded  diagnostics 
procedures  begin  with  the  simplest  non-intrusive  tests  possible  and  progress  to  the 
use  of  intrusive  tests  possible  and  progress  to  the  use  of  intrusive  test  equipment 
as  a  last  resort. 

The  nonembedded  troubleshooting  process  starts  with  the  results  of 
embedded  diagnostic  capabilities.  For  instance,  if  embedded  diagnostics  fault 
isolates  to  an  ambiguity  group  of  three  of  five  modules  in  a  system,  nonembedded 
diagnostic  should  start  the  troubleshooting  procedure  by  checking  out  the  three 
identified  modules.  This  does  not  always  occur  with  current  troubleshooting 
procedures.  Next,  the  mechanic’s  senses  (sight,  touch,  and  smell)  are  used  to 
locate  catastrophic  and/or  battle  damaged  components.  If  further  fault  isolation  is 
required,  a  Break-Out-Box  (BOB)  and  a  multimeter  are  employed  for  improved 
access  to  measure  signals  and  compare  to  good  known  values.  Finally,  if  all  other 
techniques  fail,  the  STE  is  used  to  determine  the  cause  of  failure. 
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The  following  sections  discuss  the  recommended  fixes  for  each 
diagnostic  area  presented  in  Figure  1 0,  after  considering  a  number  of  different 
alternatives. 


FIGURE  10.  Joint  Working  Group  Endorsed/Recommended  Programs 
(Short  Range) 
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A.  HULL  EMBEDDED  CAPABILITY 

1.  OBJECTIVE 

Since  mobility  is  the  key  to  a  tank’s  survival,  the  top  priority  of  hull 
diagnostics  is  to  provide  embedded  diagnostic/prognostic  capability  to  diagnose 
engine  problems.  Transmission  diagnostics  are  not  addressed  due  to  the  expense 
and  complexity  of  integrating  new  transmission  sensors  into  existing  tanks. 

2.  RECOMMENDED  FIX 

The  recommended  fix  for  engine  diagnostics  is  the  digital  Electronic 
Control  Unit  (ECU)  embedded  in  the  digital  Drivers  Instrument  Panel  (DIP),  which 
can  monitor  engine  system  signals  continuously  and  record  out-of-limit  signals 
continuously.  The  form,  fit,  and  function  permits  retrofit.  The  digital  ECU  promises 
to  provide  significant  reductions  in  O&S  cost. 

B.  TURRENT  EMBEDDED  CAPABILITY 

1.  OBJECTIVE 

Improvement  in  Stabilization  System  diagnostics  is  the  prime  objective  for 
turrent  embedded  diagnostic  capability. 

The  Laser  Range  Finder  and  Thermal  Imaging  Unit  contain  BIT  which 
fault  isolates  to  the  LRU.  The  complexity  of  the  Fire  Control  System  and 
Stabilization  System  prohibits  all  but  the  most  experienced  mechanics  (senior  NCO) 
and  Master  "D"s  from  troubleshooting  these  systems  without  the  use  of  the  STE. 
The  initial  STE  test  procedure  requires  the  STE  multiple  branch  adapter.  This 
adapter  is  cumbersome,  bulky,  and  difficult  to  connect  to  the  system.  Elimination  of 
the  multiple  branch  adapter  requirement  will  improve  diagnostic  time  by  reducing 
test  hook-up  and  run  time. 

2.  RECOMMENDED  FIX 

The  recommended  fixes  consist  of  both  short-  and  long-term  initiatives. 
For  the  short  term,  recommended  continued  development  and  fielding  of  the  STE 
multiple  entry  point  test  program.  This,  together  with  other  STE  enhancements,  will 
reduce  test  times  and  increase  soldier  acceptance  of  STE  in  troubleshooting  the 
Stabilization  System.  The  long-term  solution  is  to  incorporate  Built-In  Test  (BIT)  for 
the  Stabilization  System.  For  any  fault  not  covered  by  BIT,  recommend  use  of  the 
Break-Out-Box  and  the  Master  Fault  procedures.  STE  testing  of  the  Stabilization 
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System  will  be  retained  only  if  it  is  determined  that  BIT  and  Master  Faults  do  not 
provide  100%  fault  coverage. 

C.  HULL  AND  TURRENT  EMBEDDED  CAPABILITY  EXPANSION 

1.  OBJECTIVE 

The  recommendations  presented  in  Sections  (A)  (Hull)  and  (B)  (T urrent) 
cover  the  most  essential  subsystems  and  most  pervasive  diagnostic  problems 
encountered  in  the  tank.  However,  the  above  recommendations  do  not  cover  all 
tank  LRUs.  The  objective  of  this  section  is  to  determine  the  embedded  diagnostic 
requirements  for  the  remaining  LRUs. 

2.  RECOMMENDED  FIX 

The  recommended  approach  to  accomplishing  this  objective  is  based  on 

a  systems  engineering  methodology.  The  methodology  provides  an  objective 
means  of  determining  which  of  the  remaining  LRUs  (if  any)  are  candidates  for 
redesign  to  increase  their  embedded  diagnostic  capability. 

The  Chrysler  Huntsville  Automotive  Tank  and  Engine  Prognostics  System 
(ATEPS)  Program  proposes  a  comprehensive  redesign  of  tank  LRUs  to  improve 
reliability  by  digitizing  LRU  functions,  to  increase  BIT  capability,  and  to  provide  a 
standard  bus  architecture.  A  complete  ATEPS  design  may  be  too  expensive  to 
implement.  However,  It  may  be  cost  effective  to  selectively  implement  specific 
ATEPS  LRUs. 

The  systems  engineering  approach  to  Identifying  LRUs  for  redesign 
includes  the  following  steps: 

1 .  List  LRUs  which  cannot  be  fault  isolated  by  embedded  diagnostics 

2.  Rank  the  LRUs  by: 

-  MTBF 

-  MTTR 

-  NEOF  rate. 

3.  Select  ATEPS  LRUs  which  are  form-fit-function  replacement  for 
ranked  LRUs 

4.  Quantity  realistic  reliability  and  diagnostic  improvements  for  candidate 
ATEPS  LRUs 

5.  Perform  cost  analysis  for  each  candidate  using  the  TMDE  O&S 
Analysis  Model 

6.  Select  LRUs  for  redesign  based  on  potential  cost  avoidance. 
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D.  NONEMBEDDED  CAPABILITY 

1.  OBJECTIVE 

The  nonembedded  diagnostic  capability  improvement  objective  at  the 
down  site  and  collection  point  is  to  reduce  the  requirement  (dependency)  for  highly 
skilled  personnel.  The  approach  to  meeting  this  objective  also  addresses  the 
following  user  objectives: 

o  Reduce  requirements  for  personnel  possessing  Master  "D"  skills 

o  Reduce  size  and  weight  of  technical  manuals  and  test  equipment 

o  Reduce  dependency  on  technical  manuals  and  STE 

o  Improve  speed  of  fault  diagnosis 

o  Reduce  fault  isolation  ambiguity  groups  to  single  LRU  or  cable. 

2.  ALTERNATIVE  FIXES 

The  following  alternatives  are  candidates  for  meeting  the  nonembedded 
diagnostic  objectives: 

o  MASTER  FAULTS  PROGRAM:  The  existing  technical  manual 
troubleshooting  approach  requires  extensive  cross  referencing  which 
is  both  time  consuming  and  confusing.  Manual-based  Master  Fault 
logic  will  minimize  these  conditions  through  two  features.  First,  the 
mechanic  is  required  to  identify  the  general  problem  area,  rather  than 
select  from  a  myriad  of  starting  points  corresponding  to  hundreds  of 
symptoms.  Second,  Master  Fault  logic  leads  the  mechanic  through  all 
visual  and  aural  performance  indications,  to  isolate  the  problem  to  a 
single  LRU  or  ambiguity  group  before  test  equipment  is  required  to 
complete  troubleshooting. 

o  MASTER  "D":  The  Master  "D"  Program  provides  advanced  diagnostic 
training  to  selected  senior  mechanics.  The  Master  "D"  is  highly  skilled 
in  unit-level  diagnostics,  troubleshooting  and  Battlefield  Damage 
Assessment  of  all  primary  weapon  systems. 

o  STE  TPS  Improvements: 

1.  STE  Test  Program  Set  Multiple  Entry:  The  STE  TPS  Multiple 
Entry  is  a  demonstration  program  which  segments  the  existing  M-1 
Stabilization  Test  1400.  Multiple  entry  breaks  the  test  program 
into  segments.  Each  of  these  segments  can  be  executed 
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independently.  With  multiple  entry  points,  it  is  no  longer  necessary 
to  run  the  entire  program  from  beginning  to  end.  The  operator 
selects  the  segment  to  be  executed,  based  on  failure  symptoms. 
The  multiple  entry  Stabilization  Test  Program  is  intended  to  be  run 
without  the  multiple  branch  adapter. 

2.  STE  Test  Program  Set  Multiple  Faults:  Current  test  methodology 
requires  correction  of  each  fault  before  proceeding  with  a 
diagnostic  procedure.  The  fault  must  be  corrected  and  the  test 
reinitiated.  The  fault  may  represent  a  slight  "out-of-tolerance" 
measurement  not  significant  to  the  actual  fault  or  tank 
performance.  The  TPS  multiple  faults  methodology  will  store  all 
test  measurements  and  pass/fail  information  through  test 
completion.  The  STE  will  evaluate  all  test  results  and  provide  the 
operator  with  the  most  significant  faults  for  follow-on  corrective 
actions.  This  eliminates  current  requirement  to  restart  the  test 
program  after  each  fault  found. 

3.  STE  Test  Program  Recovery/Alert  Messages:  The  test  program 
alert  message  notifies  the  operator  when  a  measurement  has 
failed  "go  chain"  criteria  and  the  test  is  proceeding  down  a  "no-go" 
chain.  The  error  tolerant  recovery  allows  the  operator  to  retest  a 
failed  limit  measurement  when  notified  by  an  alert  message. 
When  the  retest  option  is  selected,  the  test  program  automatically 
branches  back  to  either  the  most  recent  block  of  operator  actions 
or  to  a  previously  unexecuted  diagnostic  word.  The  test  program 
then  provides  specific  recovery  instructions  before  resuming 
execution.  Alert  messages  and  error  recovery  allows  a  mechanic 
to  correct  operator  errors  without  reinitializing  the  test  program. 

4.  STE  Test  Program  Set  -20  Technical  Manual  Automatic  Follow- 
Ons:  Present  STE  testing  methodology  results  in  fault  isolation  to 
a  single  LRU  or  to  an  ambiguity  group  of  LRUs  usually  including  a 
wiring  harness.  Whenever  the  initial  test  sequence  fault  isolates  to 
an  ambiguity  group,  a  fault  message  directs  the  operator  to  the 
technical  manual  for  instructions  and  follow-on  test  procedures 
which  identifies  the  next  test  program  to  execute  for  follow-on 
testing.  Incorporation  of  the  -20  automatic  follow-ons  will  allow  the 
operator  to  continue  with  STE  testing,  including  wiring  harness 
testing  until  a  single  fault  is  isolated  without  referring  to  the 
technical  manual.  This  reduces  technical  manual  references  and 
decreases  test  time. 
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o  STE  VIDEO  TRAINING  COURSE:  The  interactive  video  disc  program 
is  intended  to  teach  STE  and  -20  technical  manual  use  by  providing  a 
hands-on  training  situation  through  interactive  dialogue  with  the  video 
disc  program. 

o  STE  HEALTH  INDICATOR  TEST  DISPLAY  (HITD):  Sections  of  the 
engine  test  program  can  be  executed  to  check  the  health  of  the 
engine  in  a  preventive  maintenance  mode  where  no  faults  are 
present.  Future  maintenance  actions  can  be  prognosticated  based  on 
STE  HITD  results. 

o  -20  TECHNICAL  MANUAL  REFORMAT:  The  -20  technical  manuals 
are  being  reformatted  in  accordance  with  MIL-M-6038.  Due  to 
elimination  of  redundant  call-outs  and  pictures,  a  35-40%  volume 
reduction  is  anticipated. 

o  MECHANICS  HELPER:  The  scope  of  this  project  is  to  perform  a 
concept  study  involving  present  expert  system  technology  and  its 
application  in  the  field  of  diagnostics  and  maintenance.  Incorporated 
in  this  study  is  an  expert  maintenance  overview  and  a  technology 
search.  This  study  will  be  used  to  examine  the  potential  of  expert 
maintenance  in  the  military  environment  and  to  determine  if  present 
expert  system  technology  i§  capable  of  supporting  military 
maintenance  requirements.  Upon  successful  completion  of  this  study, 
the  program  will  proceed  into  a  limited  hardware  demonstration.  The 
hardware  demonstration  will  provide  a  working  knowledge  of  expert 
maintenance  systems  which  would  be  applied  toward  development  of 
a  Full-Scale  Engineering  Demonstration  System  capable  of  providing 
prognostic,  diagnostic,  and  repair  information  to  the  mechanic.  The 
expert  system  should  provide  the  mechanic  access  to  a  vast 
knowledge  base  which  could  locate,  in  a  few  seconds,  the  diagnostic, 
prognostic  and  repair  information  that  would  normally  take  several 
minutes  (or  hours)  to  locate  using  current  diagnostic  techniques.  The 
expert  system  can  be  hosted  on  the  Contact  Test  Set  (CTS)  being 
developed  under  the  Intermediate  Forward  Test  Equipment  (IFTE) 
Program. 


3.  RECOMMENDED  FIX 

Recommend  the  following  approach  to  reduce  the  requirement  for  highly 
skilled  personnel. 
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Even  though  the  Master  "D"  is  the  epitome  of  highly  skilled  personnel,  he 
is  also  essential  to  the  short-term  diagnostic  solution.  His  diagnostic  techniques  do 
not  depend  on  bulky  technical  manuals  and  test  equipment  (uses  the  BOB, 
multimeter,  flow  charts,  and  wiring  diagrams).  The  Master  "D"  is  providing  and 
refining  the  Master  Fault  techniques.  This  is  an  important  step  towards  a  long-term 
solution  of  implementing  an  expert  system  based  on  Master  Fault  techniques. 

For  the  long-term,  the  Master  "D"  is  not  a  viable  program.  The  Master 
"D’s"  training  cost  are  prohibitive  (17  week  training  program).  Retention  rate  for  the 
Master  "D"  is  expected  to  be  low  due  to  promotion  to  Warrant  Officer  and  industry 
recruiting.  There  is  a  limited  pool  of  candidates  for  Master  "D"  training. 

The  STE  and  the  STE  TPS  improvements  are  endorsed  as  fixes  at  the 
Battalion  maintenance  echelon. 

The  long-term  solution  to  nonembedded  diagnostic  problems  is  the 
introduction  of  an  expert  system  such  as  the  Mechanics  Helper  to  the  diagnostic 
process.  The  expert  system  should  incorporate  the  Master  Fault  diagnostic 
procedures  tested  and  proven  by  the  Master  "D".  This  solution  eliminates  the 
reliance  on  technical  manuals,  lowers  initial  training  requirements,  and  provides  a 
built-in  tutor  for  advanced  and  sustainment  training.  Use  of  the  expert  system  by 
the  Advanced  Individual  Training  (AIT)  mechanic  guarantees  consistent 
maintenance  procedures  are  followed,  provides  consistent  diagnostic  results  and 
minimizes  intrusive  testing  at  the  down  site  and  collection  point.  Since  the  IFTE 
Contact  Test  Set  design  includes  an  expert  system  shell,  it  is  a  candidate  for 
demonstrating  the  Mechanics  Helper  capabilities. 

E.  SUMMARY 

In  summary,  the  systems  engineering  methodology  applied  to  the 
Abrams  diagnostics  improvement  resulted  in  multiple  solutions.  The  overall 
solution  involved/impacted  all  diagnostic  elements.  The  solution  focuses  on  the 
integration  of  the  diagnostic  elements  to  provide  a  comprehensive,  cohesive 
diagnostic  capability  within  the  constraints  of  an  existing  system  design  and  support 
design. 
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□f  Is  a  systems  engineering  approach  being  used  to 
modify/retrofit  the  diagnostic  capability? 


E2f  Have  all  feasible  alternatives  been  considered 
and  evaluated? 


E/  Have  emphasis  been  placed  on  correcting  the 
items  which  have  a  poor  diagnostic  history? 
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AFLC 

AFSC 

Al 

AIT 

AMC 

ATE 

ATEPS 

ATPG 

BIT 

BITE 

BOB 

C/ATLAS 

CAD 

CAE 

CALS 

CAMELOT 

CASS 

CDR 

CDRL 

CEPS 

CFE 

Cl 

CITS 

CMOS 

CND 

CNO 

COPTR 

CPCI 

CSC 

CSCI 

CSDM 

CSOM 

CTE 

CTS 

D-Level 

DBDD 


APPENDIX  B 
LIST  OF  ACRONYMS 


Air  Force  Logistics  Command 

Air  Force  Systems  Command 

Artificial  Intelligence 

Advanced  individual  Training 

Army  Materiel  Command 

Automatic  Test  Equipment 

Automotive  Tank  and  Engine  Prognostics  System 

Automatic  Test  Program  Generation 

Built-In  Test 

Built-In  Test  Equipment 

Break-Out-Box 

Common  Abbreviated  Test  Language  for  All  Systems 

Computer-Aided  Design 

Computer-Aided  Engineering 

Computer-Aided  Acquisition  &  Logistics  Support 

Computer-Aided  Measure  for  Logistic  Testability 

Consolidated  Automated  Support  System 

Critical  Design  Review 

Contract  Data  Requirements  List 

CITS  Expert  Parameter  System 

Contractor  Furnished  Equipment 

Configuration  Items 

Central  Integrated  Test  System 

Complementary  Metal  Oxide  Semi-Conductor 

Cannot  Duplicate 

Chief  of  Naval  Operation 

Controllabilfty-ObservabiBty-Predlctablllty-Testabillty  Report 

Computer  Program  Configuration  Item 

Computer  System  Component 

Computer  Software  Configuration  item 

Computer  System  Diagnostic  Manual 

Computer  Software  Operator's  Manual 

Commercial  Test  Equipment 

Contact  Test  Set 

Depot  Level 

Data  Base  Design  Document 
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DCP 

DerrWal 

DFT 

DID 

DIP 

DoD 

DoD-D 

DoD-INST 

DTA 

DSESTS 

DT&E 

ECP 

ECU 

FA 

FCA 

FD 

FEFI 

FFI 

FFD 

FI 

FI  PAD 

FIS 

FMEA 

FMECA 

FOT&E 

FRACAS 

FSD 

FSM 

FYDP 

GFE 

GIMADS 

GPETE 

HW 

HWCI 

1-Level 

1C 

ID 

IDD 

IDSS 

IFTE 

ILS 


J 


Decision  Coordinating  Paper 
Demonstration  and  Validation  (Phase) 

Design  For  Testability 

Data  Item  Description 

Drivers  Instalment  Panel 

Department  of  Defense 

DoD  Directive 

DoD  Instoiction 

Daisy  Testability  Analyzer 

Direct  Support  Electrical  Systems  Test  Set 

Development  Test  and  Evaluation 

Engineering  Change  Proposal 
Electronic  Control  Unit 

False  Alarm 

Functional  Configuration  Audit 
Fault  Detection 

Fraction  of  Erroneous  Fault  Isolation  Results 
Fraction  of  Faults  Isolated 
Fraction  of  Faults  Detected 
Fault  Isolation 

Failure  Identification,  Prevention,  and  Detection 

Fault  Isolation  System 

Failure  Modes  and  Effects  Analysis 

Failure  Modes,  Effects  and  Criticality  Analysis 

Follow-On  Test  &  Evaluation 

Failure  Repeating  Analysis  and  Corrective  Action 

Full-Scale  Development 

Firmware  Support  Manual 

Five  Year  Defense  Plan 

Government  Furnished  Equipment 
Generic  Integrated  Maintenance  Diagnostics 
General  Purpose  Electronic  Test  Equipment 

Hardware 

Hardware  Configuration  Item 

Intermediate  Level 
Integrated  Circuit 
Integrated  Diagnostics 
Interface  Design  Document 
Integrated  Diagnostics  Support  System 
Intermediate  Forward  Test  Equipment 
Integrated  Logistic  Support 
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ILSP 

Integrated  Logistic  Support  Plan 

IMIS 

Integrateo  Maintenance  Information  System 

IOT&E 

Initial  Operational  Test  &  Evaluation 

IPS 

Integrated  Program  Summary 

ITP 

Integrated  Test  Plan 

JWG 

Joint  Working  Group 

LCC 

Life  Cycle  Cost 

LOGMOD 

Logic  Modeling 

LRM 

Line  Replaceable  Module 

LRU 

Line  Replaceable  Unit 

LSA 

Logistic  Support  Analysis 

LSAP 

Logistic  Support  Analysis  Plan 

LSI 

Large  Scale  Integration 

MASTER  "D" 

Master  Diagnostician 

MATE 

Modular  Automatic  Test  Equipment 

MD 

Maintainability  Demonstration 

MIL-STD 

Military  Standard 

MNS 

Mission  Need  Statement 

MTBF 

Mean  Time  Between  Failures 

MTE 

Manual  Test  Equipment 

MTTR 

Mean  Time  To  Repair 

NCO 

Non-Commissioned  Officer 

NDI 

Non-Developmental  Items 

NEOF 

No-Evlde  nee -of-Fai  lure 

NSIA 

National  Security  Industrial  Association 

O-Level 

Organizational  Level 

OJT 

On-the-Job  Training 

OT&E 

Operational  Test  &  Evaluation 

OUSD(A) 

Office  of  the  Under  Secretary  of  Defense  (Acquisition) 

p3. 

Preplanned  Product  Improvement 

PAT&E 

Production  Acceptance  Test  &  Evaluation 

PCA 

Physical  Configuration  Audit 

PCB 

Printed  Circuit  Board 

PDR 

Preliminary  Design  Review 

PLUASL 

Prescribed  Load  List/ Authorized  Stockage  List 

PMRT 

Program  Management  ResponsfbIRty  Transfer 

PPBS 

Planning,  Programming  and  Budgeting  System 

PROD/DEP 

Production/Deployment  (Phase) 

PRR 

Production  Readiness  Review 
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RADC 

RDGT 

RF 

RFP 

RISE 

ROC 

RTOK 

SCOAP 

SCP 

SDDD 

SDR 

SEMP 

SHARP 

SIT 

SON 

SOW 

SPM 

SRA 

SRR 

SRU 

STAMP 

STD 

STE 

STE  HITD 

STLDD 

STP 

SUM 

SW 

T&E 

T 

TAH 

TBD 

TEMP 

TFOM 

Tl 

TISSS 

TM 

TMDE 

TO 

TPI 

TPS 

TRD 

TRR 


j 


Rome  Air  Development  Center 
Reliability  Development/Growth  Test 
Radio  Frequency 
Request  for  Proposal 

Readiness  Improvement  Through  System  Engineering 
Required  Operational  Capability 
Retest  OK 

Sandia  Controllability/Observability  Analysis  Program 

System  Coordinating  Paper 

Software  Detail  Design  Document 

System  Design  Review 

System  Engineering  Management  Plan 

Standard  Hardware  Acquisition  Requirement  Process 

System  Integrated  Test 

Statement  of  Need 

Statement  of  Work 

Software  Programmer’s  Manual 

Shop  Replaceable  Assembly 

System  Requirements  Review 

Shop  Replaceable  Unit 

System  Testability  Analysis  Maintenance  Program 
Software  Test  Descriptions 
Simplified  Test  Equipment 

Simplified  Test  Equipment  Health  Indicator  Test  Display 

Software  Top-Level  Design  Document 

Software  Test  Plans 

Software  User’s  Manual 

Software 

Test  &  Evaluation 
Testability 

Testability  Analysis  Handbook 
To  Be  Determined 
Test  &  Evaluation  Master  Plan 
Testability  Figure  of  Merit 
Technical  Information 

Tester  Independent  Support  Software  System 
Test  and  Maintenance 

Test,  Measurement,  and  Diagnostic  Equipment 

Technical  Orders 

Test  Program  Instruction 

Test  Program  Set 

Test  Requirements  Document 

Test  Readiness  Review 
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UUT 

Unit  Under  Test 

VHSIC 

Very  High  Speed  Integrated  Circuit 

VLSI 

Very  Large  Scale  Integration 

WBS 

Work  Breakdown  Structure 

WRA 

Weapon  Replaceable  Assembly 
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